Monitoring Cassandra with Prometheus

Introduction

Monitoring Cassandra with Prometheus can be done in various ways. Till now we have used pre-built exporters for Linux and Docker, which are non-intrusive and do a good job at monitoring.

For Cassandra, we will use something different One of the most flexible ways I have found is using the JMX Exporter rather than using an external exporter.

This blog assumes that you have a working Cassandra cluster. In case, you want to install Cassandra, head over to this link which has got very nice steps. For the purpose of this blog, we have a three-node cluster.

The blog is divided into the following sections

• Download & Install JMX exporter
• Configure JMX exporter for Cassandra
  • Configure JMX exporter
  • Configure Cassandra
  • Check metrics
• Configure Prometheus server
• Query Prometheus using PromQL
• Grafana Dashboard for Cassandra

Download & Install JMX exporter

Step 1 – JMX exporter can be downloaded easily via the maven repo. Use this link. For Linux – see below

Step 2 – Once downloaded the jar needs to be placed along with other Cassandra jars. The usual place for this would be $CASSANDRA_HOME/lib

In our Cassandra cluster, we copied to the following path. See Below

The above steps need to be performed in all the Cassandra nodes.

Configure JMX exporter for Cassandra

This is probably the most important part of this blog. But don’t worry it is not difficult!. You just need to know which files to copy or modify files. There are just two of them.

Configure JMX exporter

Cassandra exposes a truckload of metrics. You can find a whole listing of those metrics here. To enable JMX exporter to scrape metrics it needs to know a few things

• Which metrics to scrape and which “NOT” to scrape
• Rules around the naming of those metrics

Naming and filtering of the metrics can be done via regex expressions. Check out the README.md of the JMX exporter it has a nice example for naming. There is already a sample configuration file to get us started and it is available on this link

I downloaded and have stored my configuration for JMX exporter in the /etc/cassandra/conf/jmx_exporter.yml

Its now time to integrate JMX exporter to Cassandra!

Configure Cassandra

Integrating Cassandra and JMX exporter is easy and it requires only one line added! 😉 Here goes

Step – 1 – Goto the Cassandra configuration directory

In my case it is /etc/cassandra/conf

Step – 2 – Edit cassandra-env.sh

Add the following line to the cassandra configuration

The above configures JMX exporter to start as a Java agent when Cassandra starts and to show metrics on port 7070.

That’s it done! – As promised one line change! Cassandra is now ready to be bounced. It can be started using the following command

The above steps need to be performed in all the Cassandra nodes.

Step 3 – Check Metrics

Goto the following URL – http://<Cassandra_host>>:7070 – It should now be able to show you the metrics. See below.

Configure Prometheus server

For Prometheus server to scrape metrics from the Cassandra servers additional configuration needs to be added. These are put in prometheus.yml. See below

It’s time to start the Prometheus server. If you need more information on how to install/run/configure Prometheus server please refer to this blog entry.

You can check if Prometheus server is able to scrape the metrics is by navigating to Prometheus UI on http://<<prometheus-host>>:<<prometheus-port>>/graph

Prometheus can also show you if it is scrapping metrics of cassandra hosts

This would bring up something similar as below

Query Prometheus using PromQL

The metrics from JMX exporter can be queried like any other metrics. Type in Up metric. This metric is available to a jmx_exporter by default

Grafana Dashboard for Cassandra

So our Prometheus server is now able to scrape cassandra metrics. Its time to import a grafana dashboard for cassandra . For the purpose of this blog entry, I am going to import a dashboard on this link.

To import a grafana dashboard follow these steps

Step 1 – Press the + button as shown below

Step 2 – You can import by typing the id assigned by grafana website to the dashboard or directly paste the JSON. I have decided to just type in the id. See Below

Step 3 -Select the data source and folder name. Press import.

The dashboard is ready!.

Though I must add – this one may require you to check some of the charts. I had to correct the first chart. But still provides quite a lot of information to get you started.

This works across multiple nodes. You can now add/change/remove charts to suit your requirements. You can explore various individual metrics and come up with something new!

This brings us to the end of this entry. Hope you have found this entry useful. If you like it – share it. Maybe leave a comment! ?