SolusVM 2 KVM VM is not manageable and its process can't be killed via CLI

Symptoms

SVM_ERROR: failed to restart virtual server "{UUID}" without force: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats)

SVM_ERROR: failed to suspend virtual server "{UUID}": Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats)

# root@hostname:~# ps aux | grep 'Z'
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
libvirt+ 40023 19.7 0.0 0 0 ? Zl Feb25 13190:58 [qemu-system-x86] <defunct>

# root@hostname:~# ps -o pid,ppid,cmd -p 40023
PID PPID CMD
40023 1 [qemu-system-x86] <defunct>

# root@hostname:~# cat /proc/40023/cpuset
/machine.slice/machine-qemu\x2d6\x2d134f3fc9\x2dc1fc\x2d47c0\x2dbe5f\x2d0a2782a7c7ff.scope/libvirt/emulator

Cause

The state change lock held by the process indicates that the VM is undergoing or awaiting a change that has not been completed or has been interrupted, thus the system continues to maintain the lock.

Resolution

1. Kill the VM's process

# kill -9 40023

2. In case the process persists reboot the node