Dell VC Plugin – iDRAC Queued Jobs

I’ve been doing some standard maintenance on my Dell R815 hosts (upgrading BIOS, firmware, drivers etc.) and ran into some difficulty with one particular server. I use the Dell Management Plugin for VMware vCenter to carry out these operations, and besides being slow it works well.

I simply run the firmware upgrade wizard which stages the current versions to the iDRAC and schedules the deployment job to execute on boot using UEFI system services. The process automatically enters the host into maintenance mode and reboots to start the UEFI jobs.

On this particular server, the job failed and I was left in a situation where the job was scheduled, would not execute and I couldn’t see any option to delete these jobs. In my case it was a BIOS update to address the PSOD issue affecting AMD Opteron 62xx series processors.

I had a look through the logs and found the following entries (also shown in the vCenter tasks);

• • • [Firmware Update] File: R815_BIOS_F8FCX_WN32_3.0.5.EXE – Status: Failed – Message: iDRAC – A job for this update already exists in the queue and prevents the new job from being scheduled.
    • [Firmware Update] File: R815_BIOS_F8FCX_WN32_3.0.5.EXE, Status: Failed, iDRAC: 10.170.210.107, Host xxxxxxxx.xxxxxxxxxxxxx.xxx, Recommended resolution actions: iDRAC – Delete the existing job before scheduling a new update job. For Remote Services API consumers, enumerate all the jobs to find and identify the duplicate job. Then invoke the DeleteJob() method to delete this job and enable the new update job to be scheduled
    • [Firmware Update] The state of update job for file (R815_BIOS_F8FCX_WN32_3.0.5.EXE) has changed from (NEW) state to (FAILED) state.

The solution is pretty simple looking at these log entries, delete the existing job and then run the wizard again. The only problem is that there is no obvious way to delete the jobs.

To speed up the resolution I logged a support call with Dell who wanted me to reset the iDRAC to factory defaults and send them various logs (which sounded like they were clutching at straws) so I started doing my own research and found a solution using WinRM nested deep in one the Dell support forums.

Windows Remote Management (WinRM) is the Microsoft implementation of the WS-Management protocol which provides a secure way to communicate with local and remote computers using web services.

Remember to replace the $USERNAME, $PASSWORD and $IPADDRESS with your own values.

Check for existing jobs;

## Use WinRM to check for existing Jobs
winrm e cimv2/root/dcim/DCIM_LifecycleJob -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -r:https://$IPADDRESS/wsman:443 -auth:basic -encoding:utf-8

Results;

DCIM_LifecycleJob
InstanceID = JID_001371642178
JobStartTime = TIME_NA
JobStatus = Failed
JobUntilTime = TIME_NA
Message = Task for this device is already present.
MessageID = RED014
Name = update:DCIM:INSTALLED:NONPCI:159:3.0.4
PercentComplete = NA

** I had multiple entries but have only shown an excerpt to keep the post short.

Delete all existing jobs;

## Use WinRM to delete all jobs
winrm invoke DeleteJobQueue "cimv2/root/dcim/DCIM_JobService?CreationClassName=DCIM_JobService+Name=JobService+SystemName=Idrac+SystemCreationClassName=DCIM_ComputerSystem" @{JobID="JID_CLEARALL"} -r:https://$IPADDRESS/wsman -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic -format:pretty

Results;

<n1:deletejobqueue_output xml:lang="" xmlns:n1="<a href=" http:="" schemas.dmtf.org="" wbem="" wscim="" 1="" cim-schema="" 2="" root="" dcim="" dcim_jobservice"="">http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_JobService"&gt;
<n1:message>The specified job was deleted</n1:message>
<n1:messageid>SUP020</n1:messageid>
<n1:returnvalue>0</n1:returnvalue>
</n1:deletejobqueue_output>

Check for existing jobs (again);

## Use WinRM to check for existing Jobs
winrm e cimv2/root/dcim/DCIM_LifecycleJob -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -r:https://$IPADDRESS/wsman:443 -auth:basic -encoding:utf-8

Results;

DCIM_LifecycleJob
InstanceID = JID_CLEARALL
JobStartTime = TIME_NA
JobStatus = Pending
JobUntilTime = TIME_NA
Message = NA
MessageID = NA
Name = CLEARALL
PercentComplete = 0

So it’s easy when you know how, but this issue still managed to waste a lot of my time! Hopefully my post will save you from a similar experience.

 34,698 total views,  4 views today