Intel microcode issue affecting E5-2600 v2 series processors

We recently experienced a number of recurring, unexpected restarts of guest VM’s, all Windows 2008 R2 servers running MSSQL Server 2008. These VM’s were all hosted on ESXi 5.0.0 build-1489271 (update 3). All hosts in the cluster are relatively new HP DL380p Gen 8 servers, with two Intel Xeon E5-2667 v2 processors @3.3GHz.

My first thoughts were that it was something to do with the MSSQL Server 2008 as these were the only guest VM’s affected. I used the Windows debugging tools (including the correct symbols) to analyse the kernel dumps, and in all cases the probable cause was memory corruption;

Bugcheck analysis, probably caused by;

## START ##

memory_corruption ( nt!MiDeletePageTableHierarchy+9c )
######################################################

'IRQL_NOT_LESS_OR_EQUAL (a)'
An attempt was made to access a pageable (or completely invalid) address
at an interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses. If a kernel debugger is
available get the stack backtrace.

Arguments:
Arg1: fffff6fb40001de0, memory referenced
Arg2: 0000000000000000, IRQL
Arg3: 0000000000000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only
on chips which support this level of status)
Arg4: fffff800016bbacc, address which referenced memory

memory_corruption ( nt!MiBadShareCount+4c )
###########################################

'PFN_LIST_CORRUPT (4e)'
Typically caused by drivers passing bad memory descriptor lists (ie:
calling MmUnlockPages twice with the same list, etc).  If a kernel
debugger is available get the stack trace.

Arguments:
Arg1: 0000000000000099, A PTE or PFN is corrupt
Arg2: 0000000000000000, page frame number
Arg3: 0000000000000000, current page state
Arg4: 0000000000000000, 0

memory_corruption ( nt!MiResolveDemandZeroFault+2e2 )
#####################################################

'IRQL_NOT_LESS_OR_EQUAL (a)'
An attempt was made to access a pageable (or completely invalid) address
at an interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses. If a kernel debugger is
available get the stack backtrace.

Arguments:
Arg1: fffff700010818a0, memory referenced
Arg2: 0000000000000000, IRQL
Arg3: 0000000000000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only
on chips which support this level of status)
Arg4: fffff800016ce272, address which referenced memory

memory_corruption ( nt!MiRemoveWorkingSetPages+388 )
####################################################

'PAGE_FAULT_IN_NONPAGED_AREA (50)'
Invalid system memory was referenced.  This cannot be protected by
try-except, it must be protected by a Probe.  Typically the address is
just plain bad or it is pointing at freed memory.

Arguments
Arg1 fffff68000000088, memory referenced.
Arg2 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3 fffff80001855350, If non-zero, the instruction address which
referenced the bad memory address.
Arg4 0000000000000005, (reserved)

## END ##

The other common factor was the identical host hardware, so I started looking into the firmware levels and customer advisories for any known issues. One of the first things I found was a BIOS firmware update to address a cross-platform issue caused by the Intel microcode of the E5-2600 v2 series processors. This is specific to the v2 series as the E5-2600 series is not affected.

HP have addressed this in a number of recent BIOS updates which are not included in the most recent HP Service Pack for ProLiant (SPP) Version 2014.02.0(B);

Version:2014.02.10 (2 May 2014)

Version:2013.12.20 (A) (21 Jan 2014)

Version:2013.11.14 (A) (20 Dec 2013)

Version:2013.09.18 (A) (24 Sep 2013)

Version:2013.09.08 (A) (13 Aug 2013)

These excerpts above are all taken from the HP ProLiant DL380p Gen8 Server BIOS release notes, so please refer the the vendors site for a full list of fixes and enhancements.

We do have a support case open with Microsoft regarding this so it will be interesting to see what recommendations come back, but I plan to address this by updating the hosts BIOS to the current version 2014.02.10 (A) released on 2nd May 2014.

Update 13/06/2014;

Since deploying the BIOS update, we haven’t had a single instance re-occur – success!

 94,768 total views,  2 views today