What Is Airflow? – Definition and How It Works

Informational P2 what is airflow

Learn what Apache Airflow is, how DAGs and operators work, what scheduling and backfills mean, and when it is more than a workflow needs.

Apache Airflow orchestrates workflows defined as code. A workflow is a directed acyclic graph of tasks, and Airflow schedules them, respects their dependencies, retries failures, and records what happened.

1. The Core Concepts

Concept	Meaning
DAG	A workflow: tasks and the dependencies between them
Task	One unit of work
Operator	A template producing a task — run a script, call an API, execute SQL
Run	One execution of a DAG for a particular period
Scheduler	Decides what should run and when
Executor	Runs the tasks, locally or distributed

with DAG("collect", schedule="0 3 * * *", start_date=datetime(2026, 1, 1)) as dag:
    fetch = PythonOperator(task_id="fetch", python_callable=fetch_pages)
    parse = PythonOperator(task_id="parse", python_callable=parse_pages)
    load  = PythonOperator(task_id="load",  python_callable=load_records)
    fetch >> parse >> load

2. Dependencies Are the Point

Cron runs commands at times. Airflow runs tasks when their prerequisites have succeeded, which is a different and more useful guarantee.

A load task that only runs after parsing succeeded, for the same period, is tedious to arrange by hand and comes free here.

3. Runs Are Tied to Periods

Each run is associated with a period rather than merely a clock time, and tasks receive that period as context.

This is what makes backfilling coherent: running the workflow for last month's periods processes last month's data, because the period is a parameter rather than an assumption about "now".

4. Retries and Failure Handling

Tasks declare retry counts and delays. A failed task retries independently, and downstream tasks wait rather than running on incomplete input.

Failures are visible in the interface with logs attached, which is the operational difference from a cron job whose output went nowhere.

5. What It Gives Beyond a Scheduler

Visibility — run history, durations, and failure points.
Backfills for historical periods.
Alerting on failure or missed runs.
Distributed execution across workers.
Parameterisation, so one definition serves many configurations.

6. What It Costs

It is infrastructure: a scheduler, a metadata database, workers, and an interface, all to be operated and upgraded.

For a handful of independent jobs that cost is not repaid. Cron with logging and a heartbeat is genuinely adequate, and adding an orchestration platform to it is a common overreach.

7. When It Is Worth It

Tasks depend on each other and must not run out of order.
Backfills are a routine requirement.
Many workflows share components and configuration.
Failures need visibility and alerting.
Work must be distributed across machines.

8. Summary

Airflow orchestrates code-defined workflows as dependency graphs, scheduling runs per period, retrying failures, and recording history — with backfills falling out of the period model.

It earns its operational weight when dependencies, backfills, and visibility matter. Below that threshold, cron plus logging remains the proportionate answer.