Workflow Orchestration (Airflow & Prefect)

Introduction

A real data pipeline is rarely one script. It is usually a sequence of steps — pull raw data, clean it, train or update a model, publish a result — that need to run in order, on a schedule, with retries when something fails and alerts when it fails anyway. Running that by hand with a cron job works until it does not. This lesson covers the two most common Python libraries built specifically to orchestrate pipelines like this.

apache-airflow: DAG-Based Orchestration

Apache Airflow is the most widely adopted workflow orchestration tool in data engineering. A pipeline in Airflow is defined as a DAG — a directed acyclic graph of tasks, where each task depends on the ones before it, and the whole graph has no cycles. Airflow provides a scheduler that runs DAGs on a timer, a web UI to monitor runs, and built-in retry and alerting logic.

pip install apache-airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def extract_data():
    print("Pulling latest customer data from the warehouse...")

def train_model():
    print("Retraining the churn model on fresh data...")

with DAG(
    dag_id="churn_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:

    extract_task = PythonOperator(
        task_id="extract_data",
        python_callable=extract_data,
    )

    train_task = PythonOperator(
        task_id="train_model",
        python_callable=train_model,
    )

    extract_task >> train_task  # train_task runs only after extract_task succeeds

prefect: Python-Native Orchestration

Prefect solves the same core problem as Airflow — scheduling and orchestrating multi-step pipelines — but with a lighter-weight, more Python-native feel. Instead of building a DAG object explicitly, you write regular Python functions and decorate them with @task and @flow. Prefect infers the dependency graph from how you actually call the functions.

pip install prefect

from prefect import flow, task

@task
def extract_data():
    print("Pulling latest customer data from the warehouse...")
    return {"rows": 5000}

@task
def train_model(data):
    print(f"Retraining the churn model on {data['rows']} rows...")

@flow(name="churn-pipeline")
def churn_pipeline():
    data = extract_data()
    train_model(data)

if __name__ == "__main__":
    churn_pipeline()

Airflow vs Prefect

Common Mistakes

Best Practices

Frequently Asked Questions

Key Takeaways

Summary

Airflow and Prefect turn a chain of scripts into a reliable, scheduled, monitored pipeline. Next, we look at how to track what actually happened during those pipeline runs — the experiments, parameters, and metrics — with MLflow and Weights & Biases.