Event Message : Issue detected on <> : PowerPath:Path vmhbax:Cx:Tx:Lxx : could not issue async IO command: Timeout.
Most of the VMware administrators are familiar with this error in the host tasks & events. However, we ignore it since it is not causing any performance degradation or storage disconnect.
The actual reason is ESXi host with its powerpath plugin installed , is trying to synchronize the LUN where as the LUN is not responding to the host or seem to be dead.
Googling this error gave me below VMware KB where it is a known issue with powerpath 5.9 and host experiences a PSOD.
Whereas , for me Powerpath is at version 6.0 and the hosts are running vsphere 6.0.
TO RESOLVE THIS ISSUE:
- Open the vmkernel.log from the host command line. (path : /var/log/vmkernel.log)
- Locate for timeout errors as below.
Timeout error:
2016-12-09T15:28:53.667Z cpu25:33446)ScsiPath: xxxx: Command 0xf1 (cmdSN 0x768111, World xxxxx) to path vmhbax:Cx:Tx:Lxx timed out: expiry time occurs 8322ms in the past
2016-12-09T15:28:53.667Z cpu25:33446)ALERT: PowerPath:Path vmhbax:Cx:Tx:Lxx : could not issue async IO command: Timeout
Command synchronization failed: (CmdSN 0x4fb1c3 from world xxxxx to dev “naa.600xxxxxxxxxxxxxxxxxxxxxxxxx” failed)
2016-12-09T15:28:53.673Z cpu25:33446)ScsiDeviceIO: 2651: Cmd(0x439e01cb2980) 0xfe, CmdSN 0x768111 from world xxxxx to dev “naa.xxxxxxxxxxxxxxxx310ce611” failed H:0xx D:0xx P:0xx Possible sense data: 0x0 0x0 0x0.
Note: you can also use Sense decoder to decode the status code – H:0xx D:0xx P:0xx
3. Make a note of the vmhba path and naa device ID shown in the vmkernel.log.
4. Ensure the specific LUN with above NAA ID is not dead or properly masked to the ESXi host. In my case, the LUN was dead which caused the host to timeout the synchronization commands.
5. After configuration from storage end are checked and modified appropriately , rescan the host to see the changes.
virtuSolve 