Solus agent restart on SolusIO Compute Resource invokes old already executed tasks

Symptoms

• Solus agent restart on SolusIO Compute Resource makes rabbitmq to push old already executed tasks. The following message can be found in /var/log/solus/agent.log:

  CONFIG_TEXT: INFO[2021-09-30 02:54:40]/usr/src/agent/pkg/queue/queue.go:253 agent/pkg/queue.(*queue).Consume() Consumer "cr-4-worker-vm-backup-restore-0" registered! processing messages...

• Connectivity issue from Compute Resource to RabbitMQ service on SolusIO Management node. The following error can be found in /var/log/solus/agent.log on Compute Resource:

  CONFIG_TEXT: /usr/src/agent/pkg/queue/queue.go:96 agent/pkg/queue.(*queue).bindQueue() Queue "" declaration failed: Exception (504) Reason: "channel/connection is not open"

Cause

SolusIO bug with ID #SIO-3671 which will be fixed in future SolusIO updates.

Resolution

Workaround:

Note: There should not be any running tasks in SolusIO

1. Connect to Management node via SSH.
2. Backup SolusIO Management node
3. Enter rabbitmq container:

   # docker exec -ti $(docker ps --format '{{ .ID }}' -f name=solus_rabbitmq) sh

4. Fetch the list of all queues:

   # rabbitmqctl list_queues

5. Remove the queues pushed to Compute Resource with the command:

   # rabbitmqctl delete_queue QUEUE_NAME

   Replace QUEUE_NAME with the corresponding queue name.

6. Exit rabbitmq container:

   # exit

7. Restart rabbitmq with the commands:

   # docker service scale solus_rabbitmq=0
   # docker service scale solus_rabbitmq=1

8.  Connect to Compute Resource via SSH
9. Restart Solus agent with the command:

   # systemctl restart solus-agent