How to create a custom Ubuntu 20.04 image for SolusIO with Prometheus and Grafana services on board

The article describes the steps required to create a custom qcow2 image with Ubuntu 20.04 guest OS that has Prometheus monitoring tools installed along with Grafana analytics tools.

Introduction

Prometheus is a flexible open-source systems monitoring and alerting toolkit. It has a big variety of export and integration libraries that allows to configure this monitoring tool to track health for different services.

However, it has lack of data representation and analytic tools. Luckily, Grafana has a built-in option to add Prometheus server and process a data collected by it.

Go Through Installation

Prerequisites: you have a SolusIO infrastructure with management and CR node that are verified to work as expected.

Create a new Ubuntu 20.4 virtual server using pre-defined template shipped with SolusIO. It should have file-based storage type and smallest possible disk size - 3GB
Access VM via SSH and configure Prometheus itself:
# useradd --no-create-home --shell /bin/false prometheus
# mkdir /etc/prometheus
# mkdir /var/lib/prometheus
# chown prometheus:prometheus /var/lib/prometheus
# cd /tmp/
# wget https://github.com/prometheus/prometheus/releases/download/v2.25.0/prometheus-2.25.0.linux-amd64.tar.gz
# tar -xvf prometheus-2.25.0.linux-amd64.tar.gz
# cd prometheus-2.25.0.linux-amd64/
# mv console* /etc/prometheus
# mv prometheus.yml /etc/prometheus
# chown -R prometheus:prometheus /etc/prometheus
# mv prometheus /usr/local/bin/
# mv promtool /usr/local/bin/
# chown prometheus:prometheus /usr/local/bin/prometheus
# chown prometheus:prometheus /usr/local/bin/promtool

Create a service file /etc/systemd/system/prometheus.service with the following content:

CONFIG_TEXT: [Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Reload systemd manager and start/enable Prometheus:

# systemctl daemon-reload
# systemctl start prometheus
# systemctl enable prometheus
Configure Node Exporter job based on Prometheus exporter for hardware and OS metrics that allows to scrape info about the server state:
# useradd --no-create-home --shell /bin/false node_exporter
# cd /tmp/
# wget https://github.com/prometheus/node_exporter/releases/download/v1.1.1/node_exporter-1.1.1.linux-amd64.tar.gz
# tar -xvf node_exporter-1.1.1.linux-amd64.tar.gz
# cd node_exporter-1.1.1.linux-amd64/
# mv node_exporter /usr/local/bin/
# chown node_exporter:node_exporter /usr/local/bin/node_exporter
# nano /etc/systemd/system/node_exporter.service
# systemctl daemon-reload
# systemctl start node_exporter
# systemctl enable node_exporter

Create a service file /etc/systemd/system/node_exporter.service with the following content:

CONFIG_TEXT: [Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Add a corresponding 'node' job to Prometheus configuration file/etc/prometheus/prometheus.yml

WARNING: Remember that position of parameters, like number of spaces, is very important in YAML

Reload systemd manager and start/enable Node Exporter service:

# systemctl daemon-reload
# systemctl start node_exporter
# systemctl enable node_exporter
Additionally, it is possible to install and configure Alertmanager using the same approach as above.
Check that both services are up and running:

# systemctl status node_exporter
# systemctl status prometheus

Access Prometheus application via browser http://<IP_ADDRESS>:9090, check Status > Targets page and verify that both prometheus jobs has status UP
Install Grafana application:
# apt-get install -y adduser libfontconfig1
# wget https://dl.grafana.com/oss/release/grafana_7.4.2_amd64.deb
# dpkg -i grafana_7.4.2_amd64.deb
# systemctl enable --now grafana-server

Grafana is available on port 3000 - http://<IP_ADDRR>:3000 with default admin:admin credentials
Add Prometheus as a data source in Grafana using localhost:9090 URL - https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/
Import pre-defined Node Exporter Full dashboard as per Grafana documentation - https://grafana.com/docs/grafana/latest/dashboards/export-import/

Security

As of now, Prometheus has lack of security measures Therefore it makes sense to secure Prometheus end-points using firewall, ufw tool in case of Ubuntu:

# ufw allow from 127.0.0.1 to 127.0.0.1 port 9090 proto tcp
# ufw allow from 127.0.0.1 to 127.0.0.1 port 9093 proto tcp
# ufw allow from 127.0.0.1 to 127.0.0.1 port 9100 proto tcp
# ufw allow ssh
# ufw allow 53/tcp
# ufw allow 80/tcp
# ufw allow 443/tcp
# ufw allow 3000/tcp

Check that rules are added:

# ufw status

Enable ufw rules:

# ufw enable

Create Image File

Once it is verifed that Prometheus collects data and Grafana represents it, you can create an image file:

Stop VPS from SolusIO web interface
Find storage device file using Hypervisor ID of the virtual machine:
# virsh domblklist <hypervisor_id>
Example:

# virsh domblklist 8d1d19f1-a1cf-4a1f-b11c-ddc19157d77d
Target Source
------------------------------------------------------------------------
sda /var/lib/libvirt/images/438/5900ae55ce2690d4861c255ea48f4af8
hdb /usr/local/solus/iso_images/438/config.iso
Create a copy of this file:
# mv /var/lib/libvirt/images/438/5900ae55ce2690d4861c255ea48f4af8{,.source}
Convert file into a template:
# virt-sysprep -a /var/lib/libvirt/images/438/5900ae55ce2690d4861c255ea48f4af8.source
# virt-sparsify /var/lib/libvirt/images/438/5900ae55ce2690d4861c255ea48f4af8.source --convert qcow2 /tmp/ubuntu-20.4-prometheus-v3.qcow2

Application parameters

Name: most suitable name
Icon: Grafana is good enough
Cloud-init version: v2
Application image URL: link to image
Login link type: URL and link to Grafana interface via IP address (or hostname) http://{{ ip }}:3000