Thursday, November 7, 2013

No visble datastores after storage failure

Last night there was a failure in our SAN in the QA/Test VMware environment. Upon bringing the SAN back online, I observed that all the datastores have been removed from the hosts. My first thought was that this was not an complicated issue - all I would have to do is restart the hosts and when all the paths have been re-established, do a rescan for the datastores and all would be well. Unfortunately after restarting the hosts, the datastores did not come up even though I can clearly see all the paths to the LUNs. Here is a summary of the symptoms:

  • Datastore do not show up after rescanning HBAs and VMFS even though all the paths are there
  • Trying to add a new datastore will show all the LUNs, but attempting to add the datastore while preserving VMFS signatures results in an error
  • VMs all show up as inaccessible
  • LUN numbers appear to have been changed on the storage

The last point regarding the LUN number change was the crucial finding. After a bit of googling, it appears that you would need to manually remount the datastores. Here's how to do it via vCLI.

First, get the host object for the affected host (you'll have to do it one at a time or combine the following commands in a script).

$esxcli = Get-EsxCli -VMHost "vm-host1.local"


Next, list the datastores it has detected that has been unmounted.

$esxcli.storage.vmfs.snapshot.list()


The list of datastores should now be displayed. Mount each of them using the following command.

$esxcli.storage.vmfs.snapshot.mount($false,"datastore1",$null)


Repeat this process for all the affected hosts and all your datastores should be properly mounted and working again.

No comments:

Post a Comment