Airflow – Dynamic Tasks
Introduction Over the last couple of months got lots of queries around dynamic tasks. So here is a post on dynamic tasks in an airflow. Dynamic tasks is probably one of the best features of airflow. It allows you to launch airflow tasks dynamically inside an airflow DAG. A simple use case can be if you want to launch a shell script with different parameters in a list all at the same time. To make things more fun is that the list size changes all the time. This is where airflow really shines. Let’s see how If you are new to airflow – before you jump into reading this post read these posts for a getting some background Airflow getting started How to write a DAG Airflow Variables Airflow DAG – Dynamic Tasks – Example-1 Creating dynamic tasks in a DAG is pretty simple. For the purpose of this section, … Read more