Applicable to:
- SolusIO
Symptoms
- Solus agent restart on SolusIO Compute Resource makes rabbitmq to push old already executed tasks. The following message can be found in /var/log/solus/agent.log:
CONFIG_TEXT: INFO[2021-09-30 02:54:40]/usr/src/agent/pkg/queue/queue.go:253 agent/pkg/queue.(*queue).Consume() Consumer "cr-4-worker-vm-backup-restore-0" registered! processing messages...
- Connectivity issue from Compute Resource to RabbitMQ service on SolusIO Management node. The following error can be found in /var/log/solus/agent.log on Compute Resource:
CONFIG_TEXT: /usr/src/agent/pkg/queue/queue.go:96 agent/pkg/queue.(*queue).bindQueue() Queue "" declaration failed: Exception (504) Reason: "channel/connection is not open"
Cause
SolusIO bug with ID #SIO-3671 which will be fixed in future SolusIO updates.
Resolution
Workaround:
Note: There should not be any running tasks in SolusIO
- Connect to Management node via SSH.
- Backup SolusIO Management node
- Enter rabbitmq container:
# docker exec -ti $(docker ps --format '{{ .ID }}' -f name=solus_rabbitmq) sh
- Fetch the list of all queues:
# rabbitmqctl list_queues
- Remove the queues pushed to Compute Resource with the command:
# rabbitmqctl delete_queue QUEUE_NAME
Replace QUEUE_NAME with the corresponding queue name.
- Exit rabbitmq container:
# exit
- Restart rabbitmq with the commands:
# docker service scale solus_rabbitmq=0
# docker service scale solus_rabbitmq=1 - Connect to Compute Resource via SSH
- Restart Solus agent with the command:
# systemctl restart solus-agent
Comments
0 comments
Please sign in to leave a comment.