SolusIO unavailable: Error: Exception (403) Reason: “no access to this vhost”

Symptoms

• SolusIO unavailable with the following error in a browser:

  CONFIG_TEXT: CONNECTION REFUSED

• The following error is displayed in /var/log/solus/installer.log on Management node:

  CONFIG_TEXT: Error: Exception (403) Reason: “no access to this vhost”

• The following error can be found in the log /var/log/rabbitmq/crash.log inside rabbitmq docker container:

  CONFIG_TEXT: =CRASH REPORT====
  crasher:
  initial call: rabbit_vhost_process:init/1
  pid: <0.509.0>
  registered_name: []
  exception exit: {{error,{{{{case_clause,undefined},[{rabbit_queue_index,add_segment_relseq_entry,3,[{file,"src/rabbit_queue_index.erl"},{line,1164}]},{rabbit_queue_index,parse_segment_entries,3,[{file,"sr
  c/rabbit_queue_index.erl"},{line,1148}]},{rabbit_queue_index,segment_entries_foldr,3,[{file,"src/rabbit_queue_index.erl"},{line,1114}]},{rabbit_queue_index,scan_queue_segments,4,[{file,"src/rabbit_queue_index
  .erl"},{line,747}]},{rabbit_queue_index,queue_index_walker_reader,2,[{file,"src/rabbit_queue_index.erl"},{line,727}]},{rabbit_queue_index,'-queue_index_walker/1-fun-1-',2,[{file,"src/rabbit_queue_index.erl"},
  {line,710}]},{worker_pool_worker,handle_cast,2,[{file,"src/worker_pool_worker.erl"},{line,113}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]}]},{gen_server2,call,[<0.521.0>,out,infini
  ty]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",[],

Cause

corrupted rabbitmq queues

Resolution

1. Connect to SolusIO management node via SSH
2. Backup SolusIO management node.
3. Enter the rabbitmq container with the command:

   # docker exec -ti $(docker ps --format '{{ .ID }}' -f name=solus_rabbitmq) sh

4. Move the content of 'queues/' directory out the directory from the error:

   # mv /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/* /tmp/

5. Exit the container:

   # exit

6. Restart rabbitmq docker container with the commands:

   # docker service scale solus_rabbitmq=0
   # docker service scale solus_rabbitmq=1