Dell VC Plugin – iDRAC Queued Jobs

I’ve been doing some standard maintenance on my Dell R815 hosts (upgrading BIOS, firmware, drivers etc.) and ran into some difficulty with one particular server. I use the Dell Management Plugin for VMware vCenter to carry out these operations, and besides being slow it works well.

I simply run the firmware upgrade wizard which stages the current versions to the iDRAC and schedules the deployment job to execute on boot using UEFI system services. The process automatically enters the host into maintenance mode and reboots to start the UEFI jobs.

On this particular server, the job failed and I was left in a situation where the job was scheduled, would not execute and I couldn’t see any option to delete these jobs. In my case it was a BIOS update to address the PSOD issue affecting AMD Opteron 62xx series processors.

I had a look through the logs and found the following entries (also shown in the vCenter tasks);

  • [Firmware Update] File: R815_BIOS_F8FCX_WN32_3.0.5.EXE – Status: Failed – Message: iDRAC – A job for this update already exists in the queue and prevents the new job from being scheduled.
  • [Firmware Update] File: R815_BIOS_F8FCX_WN32_3.0.5.EXE, Status: Failed, iDRAC: 10.170.210.107, Host xxxxxxxx.xxxxxxxxxxxxx.xxx, Recommended resolution actions: iDRAC – Delete the existing job before scheduling a new update job. For Remote Services API consumers, enumerate all the jobs to find and identify the duplicate job. Then invoke the DeleteJob() method to delete this job and enable the new update job to be scheduled
  • [Firmware Update] The state of update job for file (R815_BIOS_F8FCX_WN32_3.0.5.EXE) has changed from (NEW) state to (FAILED) state.

The solution is pretty simple looking at these log entries, delete the existing job and then run the wizard again. The only problem is that there is no obvious way to delete the jobs.

To speed up the resolution I logged a support call with Dell who wanted me to reset the iDRAC to factory defaults and send them various logs (which sounded like they were clutching at straws) so I started doing my own research and found a solution using WinRM nested deep in one the Dell support forums.

Windows Remote Management (WinRM) is the Microsoft implementation of the WS-Management protocol which provides a secure way to communicate with local and remote computers using web services.

Remember to replace the $USERNAME, $PASSWORD and $IPADDRESS with your own values.

Check for existing jobs;
## Use WinRM to check for existing Jobs
winrm e cimv2/root/dcim/DCIM_LifecycleJob -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -r:https://$IPADDRESS/wsman:443 -auth:basic -encoding:utf-8
Results;
DCIM_LifecycleJob
    InstanceID = JID_001371642178
    JobStartTime = TIME_NA
    JobStatus = Failed
    JobUntilTime = TIME_NA
    Message = Task for this device is already present.
    MessageID = RED014
    Name = update:DCIM:INSTALLED:NONPCI:159:3.0.4
    PercentComplete = NA

** I had multiple entries but have only shown an excerpt to keep the post short.

Delete all existing jobs;
## Use WinRM to delete all jobs
winrm invoke DeleteJobQueue "cimv2/root/dcim/DCIM_JobService?CreationClassName=DCIM_JobService+Name=JobService+SystemName=Idrac+SystemCreationClassName=DCIM_ComputerSystem" @{JobID="JID_CLEARALL"} -r:https://$IPADDRESS/wsman -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic -format:pretty
Results;
<n1:DeleteJobQueue_OUTPUT xml:lang="" xmlns:n1="<a href="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_JobService">http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_JobService</a>">
<n1:Message>The specified job was deleted</n1:Message>
<n1:MessageID>SUP020</n1:MessageID>
<n1:ReturnValue>0</n1:ReturnValue>
</n1:DeleteJobQueue_OUTPUT>
Check for existing jobs (again);
## Use WinRM to check for existing Jobs
winrm e cimv2/root/dcim/DCIM_LifecycleJob -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -r:https://$IPADDRESS/wsman:443 -auth:basic -encoding:utf-8
Results;
DCIM_LifecycleJob
    InstanceID = JID_CLEARALL
    JobStartTime = TIME_NA
    JobStatus = Pending
    JobUntilTime = TIME_NA
    Message = NA
    MessageID = NA
    Name = CLEARALL
    PercentComplete = 0

So it’s easy when you know how, but this issue still managed to waste a lot of my time! Hopefully my post will save you from a similar experience.

18,457 total views, 11 views today

Share on LinkedInShare on FacebookTweet about this on TwitterShare on Google+Digg thisShare on RedditPin on PinterestEmail this to someonePrint this page

Comments

  1. Hi, thanks for your posting.
    I have the same problem with an R510.
    The DELL VC plugin starts working on the job, but then gives me the error messages.

    I tried your commands, checking for existing jobs works fine – but I receive an error at the point “delete all existing jobs”:
    — Error: Invalid use of command line. Type “winrm -?” for help. —

    The command line is correct, I verified it with “Lifecycle_Controller_2_0_Web_Services_Interface_Guide_for_Windows_v1.1.pdf”

    It’s an iDRAC 6 with FW 1.95.

    Thanks in advance,
    Chris

  2. Hi Chris,

    What version of PowerShell do you have installed? Perhaps you could try adding a set of single quotes around this section of the code ‘@{JobID=”JID_CLEARALL”}’.

    So the full syntax will look like this;

    ## Use WinRM to delete all jobs
    winrm invoke DeleteJobQueue "cimv2/root/dcim/DCIM_JobService?CreationClassName=DCIM_JobService+Name=JobService+SystemName=Idrac+SystemCreationClassName=DCIM_ComputerSystem" '@{JobID="JID_CLEARALL"}' -r:https://$IPADDRESS/wsman -u:$USERNAME -p:$PASSWORD -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic -format:pretty
    

    I suspect the original code without the single quotes will work fine from a command prompt, but you will need the quotes when executing this from PowerShell.

    Let me know if this works?

    Cheers,
    Jon

    • Thanks Jon for posting this and for the reply suggesting the single quotes. I’ve been trying to execute this via powershell on Win2k12 and had the same error.

      I’m trying to upgrade the firmware on 41 new Dell R630’s using OpenManage Essentials and after installing most of the upgrades, the Ethernet card firmware failed to install with the resulting error suggesting to remove the queued jobs.

      I have to admit, being a Linux admin, I’m new to PowerShell 🙂

      Thanks again.

    • This worked for me, thanks!

  3. Hi Jon,
    it’s an Windows Server 2008R2 with Powershell 2.0.
    I didn’t try using the cmd shell, and by adding the set of single quotes it works fine from Powershell! It’s now possible to install / upgrade the new firmware files via DELL VC plugin on my R510.

    Thank you very much!
    Cheers from Germany,

    Chris

  4. Thank you for this!!!

  5. Thanks a lot for this useful information.

    Nik

  6. awesome.
    why a bios update didn’t apply properly on 1 or 4 R630s is beyond me, but this helped heaps.

Speak Your Mind

*