What Is Dagster? – Definition and How It Works

Learn what Dagster is, how asset-oriented orchestration differs from task-oriented, and where it fits a collection pipeline.

Dagster is a data orchestrator organised around assets — the datasets a pipeline produces — rather than around tasks. You declare what should exist and how to compute it, and Dagster works out the order.

1. Assets Instead of Tasks

A task-oriented orchestrator asks: what runs, in what order?

Dagster asks: what data should exist, and what does each piece depend on?

from dagster import asset

@asset
def raw_listings(context):
    rows = fetch_listings()
    context.add_output_metadata({"rows": len(rows)})
    return rows

@asset
def clean_listings(raw_listings):
    return [normalize(r) for r in raw_listings if is_valid(r)]

@asset
def listing_report(clean_listings):
    return build_report(clean_listings)

Dependencies come from the function parameters. The graph is derived, not declared separately — so it cannot drift out of sync with the code.

2. Why That Difference Matters

For a collection pipeline the practical gain is targeted rebuilds. When a parser is fixed, the asset that depends on the parser and everything downstream needs rematerialising — while the raw fetch, which is the expensive and rate-limited part, does not.

3. Partitions

Time-partitioned assets model daily collection directly:

from dagster import asset, DailyPartitionsDefinition

daily = DailyPartitionsDefinition(start_date="2026-01-01")

@asset(partitions_def=daily)
def daily_serp(context):
    day = context.partition_key
    return collect_serp(day)

Each day is a separate materialisation with its own status, so a backfill is a range of partitions and a failed day is one partition to retry — not a whole pipeline to re-run.

4. Resources and Testability

External dependencies are injected as resources rather than constructed inside asset code:

from dagster import ConfigurableResource

class ScraperClient(ConfigurableResource):
    api_key: str
    def fetch(self, url): ...

@asset
def raw_pages(scraper: ScraperClient):
    return [scraper.fetch(u) for u in seed_urls()]

Two consequences. Credentials come from configuration rather than being embedded in code, and a test substitutes a fake resource — so asset logic is unit-testable without network access, which is unusual among orchestrators.

5. Freshness and Sensors

Freshness policies declare how current an asset should be, turning staleness into an observable condition instead of something noticed when a report looks wrong.

Sensors trigger materialisation on external events — a file arriving, a queue message, an upstream update — which suits event-driven collection better than a fixed schedule.

6. Where It Fits Collection Work

The natural mapping:

raw_pages        (partitioned by day, expensive, rate-limited)
  → parsed_records  (cheap, changes when the parser changes)
    → deduplicated
      → published_table
        → report

Because the raw layer is a separate asset, re-parsing never re-fetches. That single property is worth more than any scheduling feature when the source cannot be re-read and requests are limited.

7. Compared With Other Orchestrators

Airflow's ecosystem is larger; Dagster's model is a better fit where the pipeline's purpose is producing datasets, which is exactly the case for collection.

8. Summary

Dagster orchestrates around assets — the datasets produced — deriving the dependency graph from code, tracking lineage and materialisation, and supporting partitions, freshness policies, and injectable resources.

For collection pipelines its decisive advantage is separating expensive fetches from cheap transformations, so fixing a parser rematerialises the derived assets without re-requesting anything from the source.

Potential Related Glossary

• Web Crawling
• Data Wrangling
• Data Structure
• Data Fusion