Terraform – getting started

Introduction I recently was working on a fantastic community project. Terraform happened to be a critical piece of technology in that project and we were really out of our depth when it came to Terraform skills. That inspired me to learn Terraform and explore various aspects of AWS infrastructure provisioning and particularly around networking. So thanks to all the folks for inspiring this series of blog posts. What I discovered was interesting. Terraform is a very structured way of creating cloud resources and has no magic bullet. As a developer, I realised that it was definitely not a mythical beast. Whatever we are doing in AWS console can be pretty much be done via the terraform script there are a few exceptions but not many. That was the easy part. More than Terraform I found you will need to know how deep you want to enter the world of cloud … Read more

Airflow remote logging using AWS S3

airflow s3 image

Airflow logs are stored in the filesystem by default in $AIRFLOW_HOME/dags directory, this is also called remote logging. Airflow logs can also be easily configured to be stored on AWS S3 as well. This blog entry describes the steps required to configure Airflow to store its logs on an S3 bucket. The blog entry is divided into the following sections Introduction Create S3 Connection Configure airflow.cfg Demo Introduction For this blog entry, we are running airflow on an ubuntu server which has access to AWS S3 buckets via AWS-CLI. Note: If you are using an EC2 instance please makes sure that your instance has read-write access to S3 buckets configured. Create S3 Connection To enable remote logging in airflow, we need to make use of an airflow plugin which can be installed as a part of airflow pip install command. Please refer to this blog entry for more details. Goto … Read more

Monitoring Kafka with Prometheus

Introduction Monitoring Kafka using Prometheus is easy. In this blog entry, we will see how we can monitor the state of Kafka brokers and also how we can monitor Kafka topic lags. Till now we have used pre-built exporters for Linux, Docker and JMX exporter for Cassandra. In this blog, we will use a combination of JMX exporter and a pre-built exporter to monitor Kafka. Kafka’s metrics are well documented and are available on this link. This blog assumes that you have a working Kafka cluster. In case, you want to install Kafka, head over to this link which has got very nice steps. For the purpose of this blog, we have a three-node cluster on AWS. It has two topics. See below. The blog is divided into the following sections Download & Install JMX exporter Configure JMX exporter for Kafka Configure JMX exporter Configure Kafka Check metrics Configure Prometheus server Query … Read more

Monitoring Cassandra with Prometheus

Introduction Monitoring Cassandra with Prometheus can be done in various ways. Till now we have used pre-built exporters for Linux and Docker, which are non-intrusive and do a good job at monitoring. For Cassandra, we will use something different One of the most flexible ways I have found is using the JMX Exporter rather than using an external exporter. This blog assumes that you have a working Cassandra cluster. In case, you want to install Cassandra, head over to this link which has got very nice steps. For the purpose of this blog, we have a three-node cluster. The blog is divided into the following sections Download & Install JMX exporter Configure JMX exporter for Cassandra Configure JMX exporter Configure Cassandra Check metrics Configure Prometheus server Query Prometheus using PromQL Grafana Dashboard for Cassandra Download & Install JMX exporter Step 1 – JMX exporter can be downloaded easily via the … Read more

Monitoring Docker with Prometheus

Prometheus makes it very easy to monitor Docker containers. It is done in a completely non-intrusive way. In this blog post, we will use an exporter called cAdvisor to monitor docker containers. cAdvisor(short for container advisor) is a google project written in go lang and is quite actively maintained. The blog entry will have the following sections Setting up cAdvisor on a Linux instance Configure Prometheus server Query using PromQL Grafana dashboard for docker This blog entry assumes that there is already a Prometheus server available. To learn more about how to install a Prometheus server please refer to this blog entry. Setting up cAdvisor cAdvisor binaries can be downloaded from Github. For the use of this blog entry, we are using the latest cAdvisor 0.36.0. cAdvisor along with node_exporter makes a perfect tool for monitoring Docker containers. In case, you want to quickly get started with node_exporter read this … Read more

Monitoring Linux with Prometheus

In our previous post, we saw how to set up a simple Prometheus server. In this post, we extend it to monitor Linux servers. Monitoring of Linux servers can be done easily using Prometheus using a simple exporter provided – it is called the node exporter also called system metrics exporter. You can find a list of exporters on the Prometheus site. This blog entry is divided into Setup & Configure node exporter on Linux server Configure Prometheus server to get data from node exporter Query the data using PromQL Create a dashboard using Grafana Setup node exporter on Linux server Setting up node exporter can be done in a few ways. You could do the following Download the binaries from this link Download the source and try to build it. It’s not as difficult as it seems. Use a docker image (though not a recommended way) For our blog … Read more

Grafana dashboards – getting started!

Collection of metrics data is of limited use if you cannot analyse it. One of the ways to analyse data collected is by creating dashboards. Grafana is one of the best open-source dashboarding tools. It is fast, simple, extendable and supports many data sources. In the blog entry we will do the following Install Grafana Configure Prometheus as a data source Create a chart & a put it on a dashboard Install Grafana Grafana is available various Linux flavours. For this blog entry, we will be installing Grafana on Ubuntu. Once installed it can be started using the following command The grafana server by default starts on port 3000. All grafana settings can be easily be changed. These are available on /etc/grafana/grafana.ini. These can be edited More on that in a later blog post. Configure data source We can now login to our grafana server via a browser url – … Read more

Prometheus monitoring – getting started

This is the start of the mini-series on monitoring infrastructure using Prometheus. The aim is to give an insight into Prometheus which is an open-source tool of systems monitoring. How easy and quickly it is to set up and get value out of it. A combination of Prometheus and Grafana can be considered as an alternative to paid tools like New Relic, Data Dog. Each blog entry will provide you with the ability to incrementally add monitoring your infrastructure with minimal to no changes. We will also see how we can integrate with new communication tools like Slack and MS Teams and provide robust alerting. This blog entry is about Introduction to Prometheus Getting Prometheus up and running Exploring Prometheus UI / Demo Introduction Prometheus was developed by SoundCloud in 2012. Since then it has seen its adoption increase quite nicely. It is also a project on Cloud Native Computing … Read more

Airflow RBAC – Role-Based Access Control

Airflow version 1.10.0 onward introduced Role-Based Access Control(RBAC) as part of their security landscape. RBAC is the quickest way to get around and secure airflow. It comes with pre-built roles which makes it easy to implement. Not stopping there you could add your own roles as well. In addition, to securing various features of airflow web UI, RBAC can be used to secure access to DAGs as well. Though this feature is only available after 1.10.2. And as with any software, a few bugs popped. But this was fixed in the 1.10.7. There is an entry for a bug AIRFLOW-2694. In this blog entry, we will touch upon DAG level access as well. The blog entry is divided into few parts Enable RBAC Create users using standard roles Roles & permissions Secure DAGs with RBAC Needless to say, I am assuming you have a working airflow. If not, please head … Read more

Airflow – XCOM

Introduction Airflow XCom is used for inter-task communications. Sounds a bit complex but it is really very simple. Its implementation inside airflow is very simple and it can be used in a very easy way and needless to say it has numerous use cases. Inter-task communication is achieved by passing key-value pairs between tasks. Tasks can run on any airflow worker and need not run on the same worker. To pass information a task pushes a key-value pair. The key-value pair is then pulled by another task and utilized. This blog entry requires some knowledge of airflow. If you are just starting out. I would suggest you first get familiar with airflow. You can try this link Pushing values to XCOM Before we dive headlong into XCOM, let’s see where to find them. XCOM can be accessed via the Airflow Web UI->Admin->XComs Let’s look at a DAG below. Its sole … Read more