Apache Airflow Interview Questions

@daily is a cron preset equivalent to 0 0 * * * (midnight every day). Airflow schedules a run at the end of each data interval, not the start: the run whose interval is 2026-01-01 00:00 → 2026-01-02 00:00 is only triggered once that interval has closed, i.e. just after midnight on 2026-01-02. This 'run after the interval ends' model is the single most common source of confusion for newcomers, who expect a @daily job to fire at the start of the day. The logical_date (formerly execution_date) of that first run is the interval start, 2026-01-01, which is why the templated {{ ds }} reads as the day the data belongs to rather than the day the task ran. Option c is wrong because unpausing a DAG with catchup enabled backfills from start_date rather than running 'now'.

The >> (right-bitshift) operator is overloaded in Airflow to mean 'set downstream'. Using a list on either side fans the dependency out or in: a >> [b, c] makes both b and c downstream of a, so they become eligible to run concurrently once a succeeds; [b, c] >> d makes d downstream of both, so by the default all_success trigger rule d waits for both to complete successfully. The result is a diamond. Option b describes a linear chain (a >> b >> c >> d), which is a different graph. The list syntax is purely sugar over repeated set_downstream calls and is the idiomatic way to express fan-out/fan-in without writing each edge by hand.

all_done fires the task once every direct upstream task has reached a terminal state — success, failed, or skipped — regardless of which. That is exactly the semantics you want for teardown/cleanup work that must run no matter what happened upstream. The default all_success would skip cleanup if any upstream failed, defeating the purpose. one_success fires as soon as a single upstream succeeds (it doesn't wait for the rest), and none_failed requires that no upstream failed (skips are tolerated, but a failure blocks it) — so neither guarantees the cleanup runs after a failure. A subtle trap: with all_done the cleanup task itself reports success even when upstreams failed, so the DAG run can look green; pair it with proper alerting if upstream failure must still page someone.

Dynamic task mapping (added in Airflow 2.3) defers fan-out to run time. The DAG file is parsed without knowing the mapped count; the scheduler expands process only after its upstream (list_files) has produced its output, creating one mapped task instance per element with map_index values 0..n-1. This is what makes it different from generating tasks in a Python loop at parse time, where the count must be static and known before any run. Because expansion is runtime, the mapped collection comes from XCom under the hood, so the upstream output must be a list/dict that fits XCom's storage. A correct mental model here separates parse time (DAG structure, static) from run time (mapped instances, dynamic) — confusing the two is the classic dynamic-mapping interview trap.

Sensors poll for a condition and have two classic modes. In poke mode (the default) the sensor sits in a running task slot for its entire wait, re-checking on poke_interval — simple, but many concurrent poke sensors can exhaust worker slots and deadlock the pool, so option d is false. In reschedule mode the sensor releases its slot between checks and is re-queued each interval, trading a little scheduling overhead for far better slot economy at scale (a). A timeout is essential so a sensor that will never be satisfied fails (and can alert) rather than hanging indefinitely (c). The modern option is deferrable sensors: they hand off the wait to the lightweight triggerer process via asyncio, freeing the worker entirely and resuming only when the trigger fires (e) — the most slot-efficient choice for long waits.

True. catchup controls whether the scheduler 'fills in' the schedule intervals that elapsed between start_date and the present. With catchup=True, unpausing a DAG whose start_date is two weeks back triggers a run for each missed interval (subject to max_active_runs), which is great for backfilling historical data but a notorious foot-gun: people set a months-old start_date, unpause, and accidentally launch hundreds of runs that hammer a database or external API. The defensive defaults are to set catchup=False so only the most recent interval runs going forward, and/or pin max_active_runs to throttle concurrency. Note that catchup=False does not run 'all missed intervals as one' — it simply skips them and starts from the latest completed interval. To replay a specific historical window deliberately, you use airflow dags backfill rather than relying on catchup.

The TaskFlow API (@task) makes XCom passing implicit: a task's return value is pushed to XCom, and passing make() as the argument to show() wires up the dependency and pulls that value back as the function argument at run time. So show receives the actual dict {"rows": 42} and payload["rows"] prints 42. Option c is the pre-TaskFlow mental model — with classic operators you'd push/pull XCom manually via ti.xcom_push/xcom_pull, but the decorator handles it for you. Option d misreads @task as async; it is an ordinary synchronous Python callable wrapped as an operator. The key insight is that calling a TaskFlow function in the DAG body does not execute it inline — it registers a task and returns an XComArg placeholder that resolves to the real value only when the task runs.

BranchPythonOperator returns the task_id(s) to follow; every other directly-downstream task is skipped (not failed). So path_b is skipped. The trap is join: with the default all_success trigger rule, a skip propagates — all_success is not satisfied when an upstream is skipped, so join is skipped too, silently stranding the rest of the pipeline. This is one of the most common branching bugs. The standard fix is to set join's trigger rule to none_failed_min_one_success (or the older none_failed_or_skipped), which lets it run as long as no upstream failed and at least one succeeded — so a skipped branch is tolerated. Option d is wrong because branching produces skips, never failures.

{{ ds }} is a Jinja template macro that renders to the logical_date (the data interval start) formatted as YYYY-MM-DD — here 2026-03-14, the day the data belongs to, not the day the task physically runs. Airflow renders templated fields like bash_command against the task's runtime context before execution, so option c (literal, unrendered) is wrong: bash_command is a templated field. Option b is the classic confusion between logical date and wall-clock run time; because a @daily run for the 2026-03-14 interval fires just after midnight on 2026-03-15, the wall-clock day differs from ds, but {{ ds }} deliberately tracks the data date so your queries and file paths are idempotent and reproducible on re-run. {{ data_interval_start }} gives the same instant as a full timestamp.

The bug is a violation of idempotency, the single most important property of an Airflow task. Airflow will run a task more than once for the same logical_date — on retry after a failure, on a manual re-run, or during a backfill — and a task must produce the same end state each time. This INSERT ... SELECT blindly appends the day's rows, so the second execution duplicates them. The fix is to make the write idempotent for the partition keyed by ds: delete-then-insert that day's partition, use INSERT OVERWRITE/MERGE/upsert, or write to an ds-scoped location you replace wholesale. Option b is wrong — ds is a valid context key (and execution_date is the deprecated name); option c misreads a safe parameterized query as injection; option d is wrong because **context is the standard way a task receives the runtime context. Designing every task to be safely re-runnable is what makes retries and backfills trustworthy.

logical_date marks the data interval a run owns (its start), while the task may execute much later — typically after the interval closes. The interview point is that tasks should be written against logical_date ({{ ds }}, {{ data_interval_start }}) instead of datetime.now(), because that decouples what data a run processes from when it happens to run. That decoupling is precisely what lets retries, manual re-runs, and backfills all process the identical window and stay idempotent. A strong answer connects logical_date to reproducibility/backfill safety; a weak one just defines the term without explaining why now() is dangerous.

The core tradeoff is slot occupancy. Poke mode holds a worker slot the entire wait — fine for short waits, dangerous at scale because concurrent long pokes can starve/deadlock the pool. Reschedule mode releases the slot between checks (the sensor sits in up_for_reschedule), trading a little re-queue overhead for far better capacity use on long waits. For multi-hour waits the best modern answer is a deferrable sensor: it offloads the wait to the lightweight triggerer process via asyncio and consumes no worker slot at all. A strong answer names the slot-starvation risk of poke and recommends reschedule/deferrable for long waits.

The clean mental model is a three-layer separation: Connections store the credentials/endpoint (in the metadata DB, keyed by conn_id), Hooks are the reusable client that reads a Connection and talks to the external system, and Operators are the DAG-level task units that often delegate to Hooks. The interview-critical detail is that Hooks pull credentials from Connections rather than hard-coding them, which is how Airflow keeps secrets out of DAG source. Calling a Hook directly inside a custom task is the idiomatic escape hatch when no off-the-shelf operator fits.

A task instance moves through distinct states driven by the separation between the scheduler and the executor. It starts with no status while waiting on upstream dependencies; once the scheduler sees those dependencies satisfied (and pool/concurrency limits allow), it marks the instance scheduled. The scheduler then hands it to the executor, which queues it; a worker eventually picks it up and the instance goes running; on a clean exit it becomes success, which is what unblocks downstream tasks. Understanding this pipeline matters operationally: a task stuck in scheduled usually points to exhausted pool slots or max_active_tasks, while one stuck in queued points to the executor or workers (e.g. no Celery workers available, or a Kubernetes pod that can't be scheduled). Knowing which component owns which state is how you debug a 'why isn't my task running' incident.

retries sets how many times Airflow re-attempts a task instance after it fails before marking it failed for good — a per-task (or default_args) integer, so retries=3 allows three additional attempts. retry_delay (a timedelta) is the wait between attempts. By default that delay is constant; setting retry_exponential_backoff=True makes the wait grow after each failure (roughly doubling, capped by max_retry_delay), which is the standard pattern for transient failures against rate-limited or temporarily-overloaded external systems — backing off avoids hammering a struggling dependency. Retries only help when tasks are idempotent: a re-run must be safe to repeat, otherwise automatic retries can compound the damage of a partial first attempt.