Redis is an open-source, in-memory, stateless, distributed datastore. It is heavily used as a distributed cache, as a NoSQL for[…]
Read moreLatest Posts
Airflow – Scale-out with Redis and Celery
Introduction This post uses Redis and celery to scale-out airflow. Redis is a simple caching server and scales out quite[…]
Read moreAirflow – Scale out with RabbitMQ and Celery
Introduction Airflow Queues and workers are required if there is a need to make the airflow infrastructure more flexible and[…]
Read moreAirflow – Connections
Introduction We understand that airflow is a workflow/job orchestration engine and can execute various tasks by connecting to our environments.[…]
Read moreAirflow – Sub-DAGs
Workflows can quickly become quite complicated which makes them difficult to understand. To reduce complexity tasks can be grouped together[…]
Read moreAirflow Branch joins
In many use cases, there is a requirement of having different branches(see blog entry) in a workflow. However, these branches[…]
Read moreAirflow – Variables
In the last entry on airflow(which was many moons ago!) we had created a simple DAG and executed it. It[…]
Read moreAirflow – Branching
Introduction Branching is a useful concept when creating workflows. Simply speaking it is a way to implement if-then-else logic in[…]
Read moreSpark DataSets Performance Tuning – Resource Planning
Introduction In this post we will focus on datasets and see how they can be tuned and also see if[…]
Read moreSpark RDDs Performance Tuning – Partitioning
Now that we have gone thru resource planning, shuffling & caching to tune our spark application we have still not[…]
Read more