Unable to manage SolusIO VPS after failed backup: cannot acquire state change lock (held by monitor=remoteDispatchDomainBlockJobAbort)

Symptoms

• Unable to manage SolusIO VPS after failed backup:

  SVM_ERROR: failed to stop VM "027fc1e3-c678-468e-911c-cef45571ef5f": Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainBlockJobAbort)

• When trying to start VPS manually on the Compute Resource there is an error:

  # virsh start 027fc1e3-c678-468e-911c-cef45571ef5f
  error: Failed to start domain 027fc1e3-c678-468e-911c-cef45571ef5f
  error: Cannot access storage file '/usr/local/solus/tmp/backups/027fc1e3-c678-468e-911c-cef45571ef5f/sda-snapshot.qcow2' (as uid:107, gid:107): No such file or directory

Cause

SolusIO bug with ID #SIO-3254 which is planned to be fixed in future SolusIO updates.

Resolution

As a workaround change the disk device in the VPS config manually:

1. Connect to the corresponding Compute Resource via SSH
2. Find the path to the actual image of the VPS:
   Click here to see the instruction

   • For file-based storage the image path can be found with the command:

     # cd STORAGE/VPSID;ls -1 -d "$PWD/"*

     - STORAGE is the path of the directory where all VPSes' images are located and it can be found at SolusIO > Compute Resources > Compute Resource > Storage(by default /var/lib/libvirt/images)
     - VPSID can be found at SolusIO > Virtual Servers in ID column.
     

     For example, there is a VPS with ID 436 and with default storage path. In this case, the path can be found with the command:

     # cd /var/lib/libvirt/images/436/;ls -1 -d "$PWD/"*
     /var/lib/libvirt/images/436/e1f411a7c7889963a14b67a5521ca804

   • For LVM-based storage the image path can be found using the command:

     # a=$(lvs | grep VPSID | awk {'print $1'});b=$(lvs | grep VPSID | awk {'print $2'});echo "/dev/$b/$a"

     For example:

     # a=$(lvs | grep 436 | awk {'print $1'});b=$(lvs | grep 436 | awk {'print $2'});echo "/dev/$b/$a"
     /dev/solusvm/436

3. Edit the VPS config:

   # virsh edit UUID

   Replace UUID with actual VPS UUID. VPS UUID can be found on VPS page at SolusIO > Virtual Servers > VPS

4. Change the disk storage device to the correct one by replacing the 'source file' from the error:

   CONFIG_TEXT: ...
   <disk type='file' device='disk'>
   <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
   <source file='/usr/local/solus/tmp/backups/027fc1e3-c678-468e-911c-cef45571ef5f/sda-snapshot.qcow2'/>
   <target dev='sda' bus='scsi'/>
   ...
   

   with the actual path to VPS image from step 2:

   CONFIG_TEXT: ...
   <disk type='file' device='disk'>
   <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
   <source file='/var/lib/libvirt/images/436/e1f411a7c7889963a14b67a5521ca804'/>
   <target dev='sda' bus='scsi'/>
   ...

5. Restart libvirtd and solus-agent:

   # systemctl restart libvirtd
   # systemctl restart solus-agent