Dell VC Plugin – iDRAC Queued Jobs

I’ve been doing some standard maintenance on my Dell R815 hosts (upgrading BIOS, firmware, drivers etc.) and ran into some difficulty with one particular server. I use the Dell Management Plugin for VMware vCenter to carry out these operations, and besides being slow it works well.

I simply run the firmware upgrade wizard which stages the current versions to the iDRAC and schedules the deployment job to execute on boot using UEFI system services. The process automatically enters the host into maintenance mode and reboots to start the UEFI jobs.

On this particular server, the job failed and I was left in a situation where the job was scheduled, would not execute and I couldn’t see any option to delete these jobs. In my case it was a BIOS update to address the PSOD issue affecting AMD Opteron 62xx series processors.

I had a look through the logs and found the following entries (also shown in the vCenter tasks);

  • [Firmware Update] File: R815_BIOS_F8FCX_WN32_3.0.5.EXE – Status: Failed – Message: iDRAC – A job for this update already exists in the queue and prevents the new job from being scheduled.
  • [Firmware Update] File: R815_BIOS_F8FCX_WN32_3.0.5.EXE, Status: Failed, iDRAC: 10.170.210.107, Host xxxxxxxx.xxxxxxxxxxxxx.xxx, Recommended resolution actions: iDRAC – Delete the existing job before scheduling a new update job. For Remote Services API consumers, enumerate all the jobs to find and identify the duplicate job. Then invoke the DeleteJob() method to delete this job and enable the new update job to be scheduled
  • [Firmware Update] The state of update job for file (R815_BIOS_F8FCX_WN32_3.0.5.EXE) has changed from (NEW) state to (FAILED) state.

The solution is pretty simple looking at these log entries, delete the existing job and then run the wizard again. The only problem is that there is no obvious way to delete the jobs.

To speed up the resolution I logged a support call with Dell who wanted me to reset the iDRAC to factory defaults and send them various logs (which sounded like they were clutching at straws) so I started doing my own research and found a solution using WinRM nested deep in one the Dell support forums.

Windows Remote Management (WinRM) is the Microsoft implementation of the WS-Management protocol which provides a secure way to communicate with local and remote computers using web services.

Remember to replace the $USERNAME, $PASSWORD and $IPADDRESS with your own values.

Check for existing jobs;
## Use WinRM to check for existing Jobs
winrm e cimv2/root/dcim/DCIM_LifecycleJob -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -r:https://$IPADDRESS/wsman:443 -auth:basic -encoding:utf-8
Results;
DCIM_LifecycleJob
    InstanceID = JID_001371642178
    JobStartTime = TIME_NA
    JobStatus = Failed
    JobUntilTime = TIME_NA
    Message = Task for this device is already present.
    MessageID = RED014
    Name = update:DCIM:INSTALLED:NONPCI:159:3.0.4
    PercentComplete = NA

** I had multiple entries but have only shown an excerpt to keep the post short.

Delete all existing jobs;
## Use WinRM to delete all jobs
winrm invoke DeleteJobQueue "cimv2/root/dcim/DCIM_JobService?CreationClassName=DCIM_JobService+Name=JobService+SystemName=Idrac+SystemCreationClassName=DCIM_ComputerSystem" @{JobID="JID_CLEARALL"} -r:https://$IPADDRESS/wsman -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic -format:pretty
Results;
<n1:DeleteJobQueue_OUTPUT xml:lang="" xmlns:n1="<a href="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_JobService">http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_JobService</a>">
<n1:Message>The specified job was deleted</n1:Message>
<n1:MessageID>SUP020</n1:MessageID>
<n1:ReturnValue>0</n1:ReturnValue>
</n1:DeleteJobQueue_OUTPUT>
Check for existing jobs (again);
## Use WinRM to check for existing Jobs
winrm e cimv2/root/dcim/DCIM_LifecycleJob -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -r:https://$IPADDRESS/wsman:443 -auth:basic -encoding:utf-8
Results;
DCIM_LifecycleJob
    InstanceID = JID_CLEARALL
    JobStartTime = TIME_NA
    JobStatus = Pending
    JobUntilTime = TIME_NA
    Message = NA
    MessageID = NA
    Name = CLEARALL
    PercentComplete = 0

So it’s easy when you know how, but this issue still managed to waste a lot of my time! Hopefully my post will save you from a similar experience.

19,061 total views, 9 views today

FIX : PSOD on hosts with AMD Opteron 62xx series processors

If you’re running ESX/ESXi on either HP or Dell hosts with AMD Opteron 62xx series processors and were affected by the PSOD issue, then you will be happy to know that both vendors have now released BIOS updates to address this. My understanding is that this was actually a problem with the AMD microcode rather than a VMware, HP or Dell issue.

I was affected by this using Dell PowerEdge R815 servers immediately after upgrading the BIOS on my hosts from a mix of version 2.8.2 and version 2.9.0 to version 3.0.4. The workaround up till now was to downgrade the BIOS version on all hosts back to version 2.8.2.

Dell PowerEdge R815 Hosts

ItemValue
Processor ModelAMD Opteron(tm) Processor 6276
Processor Speed2.3 GHz
Processor Sockets4
Processor Cores per Socket16
Logical Processors64
Memory256 GB

Here are the links the vendor advisories and updates;

Interestingly Dell have only flagged the importance of this upgrade as “Recommended” – not sure about you, but I quite like my hosts to be up and running!

18,479 total views, 2 views today

Update Manager : Online depot for HP and Dell extensions

If you use Update Manager and have either HP or Dell hosts then this simple VUM configuration will save you from manually downloading VIB’s and importing them into your patch repository.

Switch to admin view, select configuration, download settings and add these two vendor URL’s into your list of download sources;

Repeat this for each download source, validate the URL, and once complete your configuration should look like this;

VUM-Download-Settings

Now you can initiate a download task, and you should see the extensions in your patch repository (see examples below) which can be attached as needed. I use custom baselines to retain control of what get’s deployed.

Dell updates;

Dell-Repository

HP updates;

HP-Repository

Update 22/10/2015;

Post updated to include the new HPE vibsdepot URL, credits go to EricB (see comments below);

“On November 1, 2015, HP vibsdepot will switch to a new host name of vibsdepot.hpe.com. Please replace any use of vibsdepot.hp.com to the new vibsdepot.hpe.com after November 1, 2015. HP vibsdepot will be referred to as HPE vibsdepot as of November 1, 2015”

126,226 total views, 30 views today