Wintel Interview Questions & Answers: BSOD -due to Memory leak on 2008 SP2

Friday, February 26, 2016

BSOD -due to Memory leak on 2008 SP2 - event id 2019

Event ID: 2019. The server was unable to allocate from the system nonpaged pool beacuse the pool was empty error on windows 2008 SP2 server .

Due to non paged pool memory exhaustion , server went to unresponsive state and BSOD occurred

Log Name: System
Source: Microsoft-Windows-Kernel-General
Date: 2/23/2016 4:44:12 PM
Event ID: 6
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: xxxxxxxxxxxxxxxxxxx
Description:
An I/O operation initiated by the Registry failed unrecoverably.The Registry could not flush hive (file): '\SystemRoot\System32\Config\SOFTWARE'.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Kernel-General" Guid="{A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D}" />
<EventID>6</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2016-02-23T15:44:12.475801500Z" />
<EventRecordID>497033</EventRecordID>
<Correlation />
<Execution ProcessID="4" ThreadID="84" />
<Channel>System</Channel>
<Computer>VM994702.perceptivecloud.com</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
<Data Name="FinalStatus">0xc000014d</Data>
<Data Name="ExtraStringLength">36</Data>
<Data Name="ExtraString">\SystemRoot\System32\Config\SOFTWARE</Data>
</EventData>
</Event>

##################################################################################

9:46:40 AM

Log Name: System
Source: srv
Date: 2/23/2016 5:00:37 PM
Event ID: 2019
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Description:
The server was unable to allocate from the system nonpaged pool because the pool was empty.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="srv" />
<EventID Qualifiers="49152">2019</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-02-23T16:00:37.916596600Z" />
<EventRecordID>497268</EventRecordID>
<Channel>System</Channel>
<Computer>VM994702.perceptivecloud.com</Computer>
<Security />
</System>
<EventData>
<Data>\Device\LanmanServer</Data>
<Binary>0000040001002C0000000000E30700C0000000009A0000C00000000000000000000000000000000001000000</Binary>
</EventData>
</Event>

9:46:42 AM

9:49:02 AM

9:49:57 AM

11:19:07 AM

C:\Program Files (x8C:\Program Files (x86)\Windows Kits\8.1\Tools\x646)\Windows Kits\8.1\Tools\x64

11:19:20 AM

C:\Program Files (x86)\Windows Kits\8.1\Tools\x64\poolmon.exe

Poolmon Result:

The top line of the output is showing that the tag “MFeB” has made 220286 allocations no frees found, resulting in 81065248 bytes of nonpaged pool use – by far the biggest consumer on the system. This looks like the likely cause of the memory leak.

11:31:42 AM

mfehdisk.sys is associated with Mcafee AV and this is know bug in Mcafee Anti virus with Patch 6

Poolmon.exe also has a few command keys that sort the output for you. Press the letter indicated below to perform the operation. It takes a few seconds for each command to work. Here is a list of a few of the commands:

P - Sorts tag list by Paged, Non-Paged, or mixed. Note that P cycles through each one.
B - Sorts tags by max byte usage.
M - Sorts tags by max byte allocation.
T - Sort tags alphabetically by tag name.
E - Display Paged, Non-paged total across bottom. Cycles through.
A - Sorts tags by allocation size.
F - Sorts tags by "frees".
S - Sorts tags by the differences of allocs and frees.
E - Display Paged, Non-paged total across bottom. Cycles through.
Q - Quit.

Environment

McAfee VirusScan Enterprise (VSE) 8.8 Patch 6

Summary

Non-page pool memory leak symptoms can manifest in numerous ways:

Bugchecks
System event 2019
RDP failures
Noticeable sluggish performance of the device
Monitoring memory and observing a climb in memory resource use that never releases

If you experience any of these symptoms after installing Patch 6, this article applies to you.

Problem

The MFeB (case sensitive) pool tag is observed to increase over time, and only when the Access Protection feature is enabled.

System Change

Recently installed VSE 8.8 with Patch 6, or upgraded to Patch 6.

Cause

A process information data structure that includes reference counting could leak memory, because the reference count is not always decremented later.

This particular pool tag leak could not be investigated through static analysis (from memory dump files generated when the pool usage was high), because the structure in question has no reference or traces for where the allocation originated.

Solution

This issue is planned for resolution in VSE 8.8 Patch 7, which is not currently released. This article will be updated when Patch 7 becomes available.

To receive email notification when this article is updated, click Subscribe on the right side of the page. You must be logged in to subscribe.

Workaround

Intel security recommends that you reboot the affected system. If you are not able to reboot at this time, you can temporarily disable Access Protection, although this is not a best practice.