Symptoms
- libvirt was downgraded from 11.10.0 to 10.10.0 while VPSs were running.
- After restarting `libvirtd`, VPSs appear as powered off in the control panel, although their QEMU processes are still running.
-
The following error is displayed for the affected VPSs:
PRODUCT_ERROR: cannot connect to virtlogd \ Cannot open log file \ Device or resource busy
Cause
Running VPSs keep the runtime state files in `/run/libvirt/qemu` created by libvirt 11.10.0. These files contain capabilities that libvirt 10.10.0 does not recognize, so the downgraded daemon cannot load them. The issue is not caused by a lack of disk space or by file permissions.
Resolution
Note: To avoid this issue in the future, shut down all VPSs before downgrading libvirt, downgrade libvirt, restart the host services or reboot the compute resource, and then start the VPSs. This way the runtime state is created by the target libvirt version.
Perform the steps below after the libvirt downgrade, as described in the SolusVM2 offline migration failed with the error: operation failed: domain is no longer running.
- Connect to the compute resource via SSH.
- Reboot the compute resource, or restart each affected VPS, so that libvirt 10.10.0 creates fresh runtime state.
- Connect to the compute resource via SSH.
-
Create a backup copy of the runtime state files:
# cp -a /run/libvirt/qemu /root/libvirt-qemu-run.bak
-
Stop the libvirt services and sockets:
# systemctl stop libvirtd libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket
-
Check the number of running QEMU processes. It must not change after the steps below:
# pgrep -fc '/usr/libexec/qemu-kvm -name guest='
-
Remove the unsupported capabilities from the runtime state files and validate the XML:
Note: Replace the `caps` array with the list of capability names that libvirt 10.10.0 does not recognize.
# for cap in "${caps[@]}"; do
grep -RFl "<flag name='$cap'>" /run/libvirt/qemu | while read -r f; do
sed -i "\\|<flag name='$cap'>|d" "$f"
done
done
# xmllint --noout /run/libvirt/qemu/*.xml -
Start the libvirt services and sockets:
# systemctl start libvirtd
# systemctl start libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket
- Check that the number of QEMU processes is the same as in step 4.
-
For each VPS UUID that has no runtime XML file in `/run/libvirt/qemu`, restart the VPS:
Note: Replace `$uuid` with the UUID of the VPS.
# pid=$(pgrep -f "/usr/libexec/qemu-kvm -name guest=$uuid,")
# virsh domstate $uuid
The output must be `shut off`
PRODUCT_WARN: Terminate the process only after you have verified that the PID belongs to the VPS with this UUID.
# kill -TERM $pid
# virsh start $uuid
# virsh domstate $uuid
The output must be `running`
Comments
Please sign in to leave a comment.