Airflow & SLA Management
Introduction Came across this interesting feature of managing SLA’s natively in airflow. Failures can happen not just by an actual failure of a task/pipeline but may a slow running task/pipeline. A slow running task/pipeline may cause downstream tasks or DAGs which depend upon it to fail. This thought got me searching and I decided to write a post about Airflow and SLAs Management. What I found was a simple solution which can help manage failures and delays. Before you jump into understanding how to manage SLAs in airflow make sure you are familiar with how to create airflow DAGs . This post is divided into the following parts Time duration is defined on a task. Airflow will monitor the performance of task/DAG when SLAs are enabled. When SLAs are breached airflow can trigger the following An email notification An action via a callback function call Note 1: SLAs checks occur … Read more