Creating custom SATP claimrules for EMC Symmetrix

As part of my migration work to EMC storage I need to create a custom SATP rule on each of my vSphere 5.0 hosts. The two obvious options are using ESXCLI from a SSH session to each host, or using PowerShell where ESXCLI is exposed using the Get-EsxCli cmdlet. The PowerShell option suits me better as I can add this claimrule to all my hosts in one go.

These are the two options;

Using ESXCLI via SSH;

## START ##

## Add custom SATP claimrule
esxcli storage nmp satp rule add -s "VMW_SATP_SYMM" -V "EMC" -M "SYMMETRIX" -P "VMW_PSP_RR" -O "iops=1" -e "EMC Symmetrix (custom rule)"

## List all SATP rules (filtered)
esxcli storage nmp satp rule list -s VMW_SATP_SYMM

## Results
Name           Device  Vendor  Model      Driver  Transport  Options  Rule Group  Claim Options  Default PSP  PSP Options  Description
-------------  ------  ------  ---------  ------  ---------  -------  ----------  -------------  -----------  -----------  ---------------------------
VMW_SATP_SYMM          EMC     SYMMETRIX                              user                       VMW_PSP_RR   iops=1       EMC Symmetrix (custom rule)
VMW_SATP_SYMM          EMC     SYMMETRIX                              system                                               EMC Symmetrix

## END ##

For the PowerShell method, make sure you check out the ESXCLI syntax;

Refer to the vSphere documentation for ESXCLI options, and Robert van den Nieuwendijk blog for the ESXCLI syntax;

#$esxcli.storage.nmp.satp.rule.add(boolean boot, string claimoption, string description, string device, string driver, boolean force, string model, string option, string psp, string pspoption, string satp, string transport, string type, string vendor)

Using ESXCLI via PowerShell;

## START ##

## Required syntax for ESXCLI (see note above)
$esxcli.storage.nmp.satp.rule.add(boolean boot, string claimoption, string description, string device, string driver, boolean force, string model, string option, string psp, string pspoption, string satp, string transport, string type, string vendor)

## This syntax translates to this (my example)
$esxcli.storage.nmp.satp.rule.add($null,$null,"EMC Symmetrix (custom rule)",$null,$null,$null,"SYMMETRIX",$null,"VMW_PSP_RR","iops=1","VMW_SATP_SYMM",$null,$null,"EMC")

## List SATP claimrules (filterd)
$esxcli.storage.nmp.satp.rule.list() | where {$_.Description -like "*Symmetrix*"} | Format-Table -AutoSize

## Results
ClaimOptions DefaultPSP Description                 Device Driver Model     Name          Options PSPOptions RuleGroup
------------ ---------- -----------                 ------ ------ -----     ----          ------- ---------- ---------
             VMW_PSP_RR EMC Symmetrix (custom rule)               SYMMETRIX VMW_SATP_SYMM         iops=1     user
                        EMC Symmetrix                             SYMMETRIX VMW_SATP_SYMM                    system   

## END ##

Now that you have the correct syntax, use PowerShell to apply this change to all your hosts. Just remember to update the scope to reflect your own environment.

Adding SATP claimrule to multiple hosts;

## START ##

Clear-Host

## Connect to vCenter
Connect-VIServer -Server 'myvcenter.fqdn'
Write-host ""

## Get list of hosts that you want to create the SATP claimrule on
$scope = Get-Datacenter 'MY-DATACENTER' | Get-Cluster * | Get-VMHost * | Sort-Object Name

## Action for each of the hosts in scope
Foreach ($esx in $scope){

  Write-Host $esx -ForegroundColor Yellow

  ## Exposes the ESXCLI functionality
  $esxcli = Get-EsxCli -VMHost $esx

  ## Create user defined SATP rule for EMC/Symmetrix (Vendor/Model)
  $esxcli.storage.nmp.satp.rule.add($null,$null,"EMC Symmetrix (custom rule)",$null,$null,$null,"SYMMETRIX",$null,"VMW_PSP_RR","iops=1","VMW_SATP_SYMM",$null,$null,"EMC")

  ## List SATP rule with "Symmetrix" in the description
  $esxcli.storage.nmp.satp.rule.list() | where {$_.Description -like "*Symmetrix*"} | Format-Table -AutoSize

}

## Disconnect from vCenter
Disconnect-VIServer -Server * -Force -Confirm:$false 

## END ##

If you just want to list the rules across all your hosts then comment out line 21 (which creates the claimrule) and execute the script again. The formatting is not the best in this instance, but it’s a quick way of validating what you’ve just done.

12,626 total views, no views today

EMC Symmetrix VMAX 40K testing on vSphere 5.0

We are in the process of migrating from HDS to EMC storage and I have been testing our Symmetrix VMAX 40K on vSphere 5.0. This has been an interesting journey and has highlighted that although similar concepts (ie. block storage with FC connectivity), storage arrays differ and need careful implementation if you want to get the best performance from your infrastructure.

This post will cover my testing with this specific storage array and hopefully prompt some feedback on other implementations. Perhaps it will help identify any obvious areas that I have missed and need to address? Either way, some feedback would be awesome.

In terms of storage presentation to the hosts (HP DL380p Gen 8 servers), I used 2x single port QLE2560 HBA’s, each connected at 4GB to Brocade FC switches with 2x paths to each LUN (4 in total). The LUNs were configured as striped META’s, each 2TB in size.

For my performance tests, I ran a series of iometer access specifications and graphed the results in Excel for an easy side by side comparison.

Iometer Access Specifications;

Access Specification Transfer SizeRead / WriteRandom / SequentialAligned on
Max Throughput-100%Read32K100% / 0%0% / 100%32K
RealLife-60%Rand-65%Read8K65% / 35%60% / 40%8K
Max Throughput-50%Read32K50% / 50%0% / 100%32K
Random-8k-70%Read8K70% / 30%100% / 0%8K

All tests were run in 30 second intervals, increasing the number of outstanding IO requests, using exponential stepping to the power of 2, up to a maximum queue depth of 512 outstanding IOs.

The workload was initially placed on a single ESXi host with a single worker thread (to get a baseline), and then scaled out to multiple ESXi hosts with multiple worker threads. The guest VM’s were not optimized in any way and had a single LSI Logic SAS controller (Windows 2008 R2 Standard Edition).

For my first baseline, I used standard NMP with all the defaults;

# START #

~ # esxcli storage nmp device list -d naa.60000970000295700663533030383446
naa.60000970000295700663533030383446
   Device Display Name: EMC Fibre Channel Disk (naa.60000970000295700663533030383446)
   Storage Array Type: VMW_SATP_SYMM
   Storage Array Type Device Config: SATP VMW_SATP_SYMM does not support device configuration.
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=3: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba3:C0:T3:L1, vmhba3:C0:T4:L1, vmhba2:C0:T4:L1, vmhba2:C0:T3:L1

~ # esxcli storage nmp psp roundrobin deviceconfig get -d naa.60000970000295700663533030383446
   Byte Limit: 10485760
   Device: naa.60000970000295700663533030383446
   IOOperation Limit: 1000
   Limit Type: Default
   Use Active Unoptimized Paths: false

# END #

Here are the results;

NMP Results (policy=rr,iops=1000);

EMC_SYMM_VMAX_40K_NMP_RR_IOPS_1000

For my next test, I changed the IO operations limit from the default 1000 to 1, as recommended in the EMC document (see pages 82-83);

Using EMC Symmetrix Storage in VMware vSphere Environments

# START #

~ # esxcli storage nmp satp rule add -s "VMW_SATP_SYMM" -V "EMC" -M "SYMMETRIX" -P "VMW_PSP_RR" -O "iops=1"

~ # esxcli storage nmp satp rule list -s VMW_SATP_SYMM
Name           Device  Vendor  Model      Driver  Transport  Options  Rule Group  Claim Options  Default PSP  PSP Options  Description
-------------  ------  ------  ---------  ------  ---------  -------  ----------  -------------  -----------  -----------  -------------
VMW_SATP_SYMM          EMC     SYMMETRIX                              user                       VMW_PSP_RR   iops=1
VMW_SATP_SYMM          EMC     SYMMETRIX                              system                                               EMC Symmetrix

# END #

I rebooted my hosts at this point, and confirmed that the device had been claimed correctly;

# START #

~ # esxcli storage nmp device list 
naa.60000970000295700663533030383446
   Device Display Name: EMC Fibre Channel Disk (naa.60000970000295700663533030383446)
   Storage Array Type: VMW_SATP_SYMM
   Storage Array Type Device Config: SATP VMW_SATP_SYMM does not support device configuration.
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba2:C0:T3:L1, vmhba3:C0:T1:L1, vmhba2:C0:T1:L1, vmhba3:C0:T3:L1

# END #

Here are the new performance results using the {policy=rr,iops=1} device configuration.

NMP Results (policy=rr,iops=1);

EMC_SYMM_VMAX_40K_NMP_RR_IOPS_1

WOW, that single change increased performance by a factor of 4X using a single worker thread! Throughput increased from just over 300 MB/s to 800 MB/s, operations increased from 10,000 IOPS to 40,000 IOPS and average latency dropped from 70ms to 25ms at a MAX queue depth of 512.

The results were similar in the scale out tests, with the same observation that guest CPU utilisation increased with the additional workload that it was able to process.

I then implemented EMC PowerPath/VE to see how this compared to NMP in the identical iometer tests. My assumption was that PowerPath/VE would far out perform NMP and the cost would be easily justifiable by the performance gains.

Interestingly, the results were similar to NMP with the IO operations set to 1 which has made it hard to sell to management. I understand the benefits that it offers over NMP, but perhaps these will only become apparent when we ramp up the workload and need the extra smarts behind the workload.

PowerPath/VE Results;

EMC_SYMM_VMAX_40K_MPP_POWERPATH_VE

These are obviously very simple tests, but it’s incredible how much performance can change by simply reading vendor recommendations and testing this in your own environment.

19,202 total views, 12 views today