In Prefect, how would you cap concurrent calls that many flows make to one shared API?

## Why per-flow limits are the wrong layer The constraint you are protecting belongs to the **API**, not to any one flow. If each flow limits itself to four concurrent calls, the vendor sees four times however many flows happen to run at once — a number that changes every time someone adds a deployment. Any cap that is going to hold has to be expressed once, somewhere both flows can see. In Prefect that is the concurrency system, which stores counters server-side so every worker, pod and flow run consults the same accounting. ## Mechanism one: tag-based task concurrency limits Prefect task runs carry tags, and a tag can be given a maximum number of simultaneously Running task runs: ```python @task(tags=["vendor-api"], retries=3, retry_delay_seconds=10) def fetch(customer_id: str): ... ``` With a limit of 5 on `vendor-api`, at most five task runs bearing that tag execute at once, no matter which flow submitted them or which worker picked them up. Runs beyond the limit wait for a slot rather than executing. The slot is held for as long as the task run is in a Running state and released when it leaves. This is the right tool when the **task is the unit of contention** — one task, one call, one slot. It is a blunt instrument when a task does an API call and then thirty seconds of local work: the slot is held for the whole task, so your effective concurrency against the API is lower than the number you set. ## Mechanism two: global concurrency limits and rate limits Global concurrency limits are named, server-side counters you acquire explicitly around any block of code: ```python from prefect.concurrency.sync import concurrency, rate_limit @task def fetch(customer_id: str): rows = prepare(customer_id) # unthrottled with concurrency("vendor-api", occupy=1): payload = call_vendor(rows) # only this holds a slot return transform(payload) # unthrottled ``` Two things this buys you over tags. First, precision: only the contended section occupies the slot. Second, weighting — `occupy` lets an expensive operation take several slots, so a bulk call and a single-row call can share one budget honestly. `rate_limit(...)` addresses the other shape of vendor constraint: not "how many at once" but "how many per second", pacing releases rather than counting concurrent holders. An async variant of the same API exists for async flows. ## Mechanism three: the infrastructure layer Work pools and deployments can also cap how many runs execute concurrently. That is a coarser, infrastructure-shaped control — it protects your cluster and your budget rather than the vendor — and it is the wrong knob for this problem, because it throttles whole flow runs regardless of whether they touch the API at all. ## Choosing, as a design decision - Contention is the entire task, and the tag is a natural label for it → **tag limit**. - Contention is one call inside a larger task, or different calls cost different amounts → **global limit with `occupy`**. - The vendor publishes requests-per-second rather than concurrency → **rate limit**. - You are protecting your own compute or spend → **work pool / deployment concurrency**. Often the honest answer is two of them: a rate limit for the vendor's published quota and a work-pool cap so a backfill cannot flood your cluster. ## The operational edges a lead is expected to name **Leaked slots.** A run killed by its infrastructure never leaves Running cleanly, so its slot can stay occupied. The queue then drains more slowly than the limit implies, or stalls entirely. Know that slots can be reset, and monitor for a limit sitting pinned at its maximum with nothing actually running. **Self-deadlock.** If a flow acquires a slot and then waits on subflows or tasks that need the same limit, the parent holds the resource its children need and nothing progresses. Acquire around leaves, not around orchestration. **Queueing is not free.** Waiting task runs still occupy a worker's attention or sit in a waiting state; a very low limit with a very wide fan-out turns into a long tail of slow runs and timeouts that look like failures. Set `timeout_seconds` deliberately so a starved run fails visibly instead of hanging. **Throttling is not error handling.** A shared cap reduces the chance of a 429 but does not eliminate it — other consumers exist, and vendor limits change. Keep `retries` with backoff on the calling task, and treat the throttle as capacity shaping rather than correctness. **Governance.** Because these limits are workspace-level, they are shared state: one team raising a limit affects another team's throughput. Decide who owns each named limit, document what constraint it encodes and where the number came from, and review it when the vendor contract changes — an undocumented number that nobody dares touch is the usual end state.