Terraform – Provisioners

Introduction Terraform Provisioners allow you to execute custom scripts/code on the remote machine which is being provisioned or something locally on which the terraform code is being executed. There are various provisioners which are available in terraform. In this blog, we will look at the generic type of provisioners – file, local and remote provisioners. The blog builds from this previous blog and is divided into the following sections. All the code for these sections are available on the github as well. The link is provided at the end of each section. File Provisioner File provisioner allows you to transfer files locally on which terraform code is being executed to the resource which is being provisioned. For example, you want to copy a configuration file or directory from the local to your EC2 instance. This can be easily accomplished by File provisioner. Here is the example of the instance.tf which … Read more

Terraform – State

Introduction Terraform state is something which is usually tackled quite later in the game when folks have become quite comfortable with writing terraform code. But, we are going to do it much earlier :). You will see that it helps you visualise your code a lot better and have more interesting/innovative ideas. In this blog entry, we discuss terraform state by breaking it down into smaller chunks. You can find all the code for this blog here. Look for blog-5. So let’s start off…. What is terraform state? Terraform state is a record of any infrastructure or configuration which has been provisioned by terraform. Simple as that. So, whenever you add/change any part of the terraform code and apply it, terraform will compare it with the state and apply only the required changes. It does not execute the whole code which makes it pretty performant if you have large infrastructures. … Read more

Terraform – Variables

Introduction Till now we have pretty much kept the code written pretty simple and straight forward. In the real world, it is hardly ever is. Similar to other languages terraform also provides different types of variables and they can be used in many different ways. In this blog, we will introduce various types of terraform variables and see how we can use them inside the terraform code. The blog post is divided into the following sections As you can see it makes a pretty long read. But it will be very helpful in your projects. The various ways of passing variables to terraform helps you manage What is source controlled/what is NOT! How are secrets managed? How is terraform code executed? This blog entry creates the same infra-structure which was built in the previous blog post but uses variables to make it more reusable. Before we get started all the … Read more

Terraform – Security Groups & EC2 instances

Introduction In this blog post, we will see how we can create AWS security groups, EC2 instances and see how they can both be configured together. It is simple and can be done quickly via Terraform. This post is in some sense continuation of the previous post on Security Groups. It introduces how you can configure your EC2 and also introduces an additional security feature – SSH keys. This blog entry is divided into the following sections In addition, to this, we would be using IntelliJ IDE with Terraform plugin for development. All the code for these blogs is now available on GitHub in the following repo. Before jumping into this blog, please make sure you have gone thru the Terraform – Getting Started and Terraform and Security Groups. Create AWS security group So let’s get started. Let’s first create a security group for our blog post. It is similar … Read more

Terraform – AWS Security Groups

Introduction This blog talks about the mythical beast called AWS Security groups and how to implement them using terraform. By the end of this blog, you would be able to do describe security groups, how you can implement them using AWS console, and also via terraform. Once you understand what security groups are, implementing them using AWS console or Terraform is a fairly simple exercise. If you are new to terraform please have a look at the Terraform – getting started blog entry first. This blog entry is divided into the following sections What are AWS Security Groups AWS Security Group as the name alludes has to do with security. It is a virtual firewall for your AWS service and controls inbound and outbound traffic. Security groups are based on simple rules. These rules defined based on three things Protocol Port IP address range Security groups are like a ring … Read more

Terraform – getting started

Introduction I recently was working on a fantastic community project. Terraform happened to be a critical piece of technology in that project and we were really out of our depth when it came to Terraform skills. That inspired me to learn Terraform and explore various aspects of AWS infrastructure provisioning and particularly around networking. So thanks to all the folks for inspiring this series of blog posts. What I discovered was interesting. Terraform is a very structured way of creating cloud resources and has no magic bullet. As a developer, I realised that it was definitely not a mythical beast. Whatever we are doing in AWS console can be pretty much be done via the terraform script there are a few exceptions but not many. That was the easy part. More than Terraform I found you will need to know how deep you want to enter the world of cloud … Read more

Airflow remote logging using AWS S3

airflow s3 image

Airflow logs are stored in the filesystem by default in $AIRFLOW_HOME/dags directory, this is also called remote logging. Airflow logs can also be easily configured to be stored on AWS S3 as well. This blog entry describes the steps required to configure Airflow to store its logs on an S3 bucket. The blog entry is divided into the following sections Introduction Create S3 Connection Configure airflow.cfg Demo Introduction For this blog entry, we are running airflow on an ubuntu server which has access to AWS S3 buckets via AWS-CLI. Note: If you are using an EC2 instance please makes sure that your instance has read-write access to S3 buckets configured. Create S3 Connection To enable remote logging in airflow, we need to make use of an airflow plugin which can be installed as a part of airflow pip install command. Please refer to this blog entry for more details. Goto … Read more

Monitoring Kafka with Prometheus

Introduction Monitoring Kafka using Prometheus is easy. In this blog entry, we will see how we can monitor the state of Kafka brokers and also how we can monitor Kafka topic lags. Till now we have used pre-built exporters for Linux, Docker and JMX exporter for Cassandra. In this blog, we will use a combination of JMX exporter and a pre-built exporter to monitor Kafka. Kafka’s metrics are well documented and are available on this link. This blog assumes that you have a working Kafka cluster. In case, you want to install Kafka, head over to this link which has got very nice steps. For the purpose of this blog, we have a three-node cluster on AWS. It has two topics. See below. The blog is divided into the following sections Download & Install JMX exporter Configure JMX exporter for Kafka Configure JMX exporter Configure Kafka Check metrics Configure Prometheus server Query … Read more

Monitoring Cassandra with Prometheus

Introduction Monitoring Cassandra with Prometheus can be done in various ways. Till now we have used pre-built exporters for Linux and Docker, which are non-intrusive and do a good job at monitoring. For Cassandra, we will use something different One of the most flexible ways I have found is using the JMX Exporter rather than using an external exporter. This blog assumes that you have a working Cassandra cluster. In case, you want to install Cassandra, head over to this link which has got very nice steps. For the purpose of this blog, we have a three-node cluster. The blog is divided into the following sections Download & Install JMX exporter Configure JMX exporter for Cassandra Configure JMX exporter Configure Cassandra Check metrics Configure Prometheus server Query Prometheus using PromQL Grafana Dashboard for Cassandra Download & Install JMX exporter Step 1 – JMX exporter can be downloaded easily via the … Read more

Monitoring Docker with Prometheus

Prometheus makes it very easy to monitor Docker containers. It is done in a completely non-intrusive way. In this blog post, we will use an exporter called cAdvisor to monitor docker containers. cAdvisor(short for container advisor) is a google project written in go lang and is quite actively maintained. The blog entry will have the following sections Setting up cAdvisor on a Linux instance Configure Prometheus server Query using PromQL Grafana dashboard for docker This blog entry assumes that there is already a Prometheus server available. To learn more about how to install a Prometheus server please refer to this blog entry. Setting up cAdvisor cAdvisor binaries can be downloaded from Github. For the use of this blog entry, we are using the latest cAdvisor 0.36.0. cAdvisor along with node_exporter makes a perfect tool for monitoring Docker containers. In case, you want to quickly get started with node_exporter read this … Read more