Monitoring Linux with Prometheus

In our previous post, we saw how to set up a simple Prometheus server. In this post, we extend it to monitor Linux servers. Monitoring of Linux servers can be done easily using Prometheus using a simple exporter provided – it is called the node exporter also called system metrics exporter. You can find a list of exporters on the Prometheus site. This blog entry is divided into

Setup & Configure node exporter on Linux server
Configure Prometheus server to get data from node exporter
Query the data using PromQL
Create a dashboard using Grafana

Setup node exporter on Linux server

Setting up node exporter can be done in a few ways. You could do the following

Download the binaries from this link
Download the source and try to build it. It’s not as difficult as it seems.
Use a docker image (though not a recommended way)

For our blog entry, we will directly download the binaries on our EC2 Linux server.

Step – 1.1 Download node exporter on EC2 server

Node exporter can be downloaded using the following command

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.0-rc.0/node_exporter-1.0.0-rc.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.0.0-rc.0.linux-amd64.tar.gz

Node exporter by default enables various collectors and these are available in the README.md. These can be disabled or enabled by passing parameters at runtime. The only thing to remember is perf collector is not enabled by default due to the kernel and/or security settings.

Node exporter can be run simply using this command. This would run the node exporter and expose the metrics on port 8000.

./node_exporter --web.listen-address=":8000"

Not the greatest ways of running a metrics collector!! but the node exporter is now exposing metrics which can now be scrapped by the prometheus server.

However, you could create a systemd script as well. What if you use something else. Well, worry not. You can find a host of scripts on this path in node exporter GitHub repo so take a pick!. In production environments, you would probably run it as a service which starts at startup time.

Step – 1.2 Check the metrics

The metrics can be accessed using the following URL – Http://<server-name>>:8000/metrics. You can use

Configure Prometheus server

We need to add the following to our prometheus.yml file.

- job_name: 'my_ec2_monitor'
static_configs:
- targets: ['ec2-3-8-6-50.eu-west-2.compute.amazonaws.com:8000']

The above basically says you will find the metrics on the port 8000. The default path is /metrics. Hence no mention of the path. My prometheus.yml now looks like this. The red box highlights what I have added.

It is time to start the Prometheus server. If you want to know how to start it. Please see the getting started blog entry.

Query Linux metrics using PromQL

Its now time to see if all works. Navigate to the Prometheus Web UI. See below.

If you start typing in the expression you would be able to see the metrics being collected. See below

Below shows the cpu metrics for the linux server.

Let’s filter for the cpu when it is in the iowait mode. See below

Observe, how we have added a filter mode=’iowait’ and when we execute it only two rows are returned. Now, let’s see the graph

Setup a Grafana dashboard

If you want to quickly set up/install Grafana please refer to this blog entry. In this section, we will only import a prebuilt dashboard. There are loads of open source dashboards available for metrics based on node exporter and we are going to use this open-source dashboard.

Grafana being open source – there are many prebuilt dashboards which are already available. This means that we can jump-start our monitoring by reusing these open source dashboard. Node exporter is no different. Here is a link on grafana website to see a list of all open-source node exporter dashboards. I have used a dashboard from the grafana site for this blog entry.

It is available on this link, this is an actively maintained dashboard so you may find some changes to mine. Also, it can be used to monitor multiple instances.

To import a grafana dashboard follow these steps

Step 1 – Press the + button as shown below

Step 2 – You can import by typing the id assigned by grafana website to the dashboard or directly paste the JSON. I have decided to just type in the id. See Below

Step 3 – Press Enter and you should see like this below

Step 4 – Select the data source and folder name. Press import.

The dashboard is ready! Now that is some jump start 🙂

You can now add/change/remove charts to suit your requirements. This brings us to the end of this entry. Hope you have found this entry useful. If you like it – share it. Maybe leave a comment! 🙂