What Is Prefect? – Definition and How It Works

Learn what Prefect is, how flows and tasks turn Python functions into observable workflows, and where it suits collection.

Prefect is a Python workflow orchestrator built around decorating ordinary functions. A @flow is a workflow, a @task is a retryable unit inside it, and the structure is whatever your Python control flow does at runtime.

1. The Basic Shape

from prefect import flow, task

@task(retries=3, retry_delay_seconds=[5, 20, 60], timeout_seconds=90)
def fetch(url: str) -> str:
    return http_get(url)

@task
def parse(html: str) -> dict:
    return extract(html)

@flow(name="collect-listings", log_prints=True)
def collect(urls: list[str]):
    pages = fetch.map(urls)
    records = parse.map(pages)
    store([r for r in records if r])

Retries, backoff, and timeouts are task configuration rather than code you write — which removes the most commonly hand-rolled and most commonly wrong part of a collection script.

2. Dynamic Workflows

Prefect's distinguishing property is that the graph is discovered as the flow runs, not declared in advance:

@flow
def crawl(seed: str, max_pages: int = 500):
    seen, queue, out = set(), [seed], []
    while queue and len(seen) < max_pages:
        batch = [u for u in queue[:20] if u not in seen]
        queue = queue[20:]
        seen.update(batch)
        for page in fetch.map(batch):
            record, links = parse_page(page)
            out.append(record)
            queue.extend(l for l in links if l not in seen)
    return out

A static-DAG orchestrator cannot express this directly. Crawling is inherently dynamic — the work is discovered by doing it — so a loop that expands its own queue is the natural formulation, and Prefect runs it as-is.

3. Retries Where They Belong

Task-level retry with a per-attempt delay list is the feature that matters most for collection. Compare:

• Without it — every fetch function grows its own retry loop, backoff, and jitter, each slightly different.
• With it — one decorator, consistent behaviour, and every attempt visible in the run history.

That last part is what makes it more than convenience: a failure that retried twice before succeeding is recorded, so a source degrading over time is visible before it fails outright.

4. Concurrency and Rate Limits

.map runs tasks concurrently with a task runner, and global concurrency limits cap parallelism per tag:

@task(tags=["example.com"])
def fetch(url): ...

A concurrency limit on that tag bounds simultaneous requests to one domain regardless of how many flows are running — which is politeness enforced centrally rather than per script, and the only version of it that survives multiple pipelines sharing a target.

5. Deployment Model

A flow is deployed with a schedule and a work pool; workers poll for runs and execute them. Workers run wherever they are installed — a container, a VM, a Kubernetes cluster — while scheduling and observability are centralised.

For collection, that separation is useful: workers can sit in the network location or region the target requires, while the control plane stays in one place.

6. Compared With Other Orchestrators

Prefect is the lightest to adopt from existing Python, since a working script becomes a flow by adding decorators. Dagster's asset model gives better lineage; Airflow's ecosystem is larger.

7. Practical Notes

• Set timeout_seconds on every network task. Without it a hung request holds a worker indefinitely.
• Return data, not side effects, from tasks where possible — it makes caching and retries meaningful.
• Use task caching for expensive idempotent fetches, so a re-run of a partially failed flow does not refetch.
• Tag by target domain and set concurrency limits there, not in the flow.
• Keep flows importable so they can be tested by calling them directly.
• Log the counts that matter — pages fetched, records extracted, records rejected — because a flow that succeeds while collecting nothing is the failure worth catching.

8. Summary

Prefect turns Python functions into orchestrated flows and tasks with declarative retries, timeouts, and concurrency limits, and its runtime-discovered graph expresses crawls that a static DAG cannot.

Its practical value for collection is centralised retry behaviour and per-domain concurrency limits that hold across every pipeline — provided every network task has a timeout and the flow logs enough counts to distinguish success from silent emptiness.

Potential Related Glossary

• Web Crawling
• Concurrency vs Parallelism
• Data Wrangling
• Data Structure