When would you attach an HTTP pull consumer to a Cloudflare queue instead of a consumer Worker, and how does it pull and acknowledge messages?
answer
- consumption outside Workers
- one consumer type per queue
- a lease per message
- short polling, not long
basics
~20 sUse an HTTP pull consumer when processing runs outside Workers or must set its own pace. Enable it with wrangler queues consumer http add, POST to the queue's messages/pull endpoint, then send each lease_id to messages/ack before the visibility timeout.
solid answer
~50 sA push consumer is a Worker whose `queue()` handler Cloudflare invokes and scales automatically. A pull consumer is any HTTP client, for example a GPU service in another data centre or a worker that must not exceed a downstream rate. A queue has one consumer type, so an existing Worker consumer is removed before `wrangler queues consumer http add orders`; setting `type` to `http_pull` in the Wrangler file is rejected. The client uses an API token with Queues read and write permissions and POSTs to `/accounts/{account_id}/queues/{queue_id}/messages/pull` with `batch_size` (default 5, maximum 100) and `visibility_timeout_ms` (default 30 seconds, maximum 12 hours). Each message carries a `lease_id`; the client POSTs `acks` and `retries` lists of lease IDs to `/messages/ack`. Unacknowledged messages return to the queue when the timeout expires. Pulls are short polls, and a pull on an empty queue still costs a read.
code
bash · 11 linesnpx wrangler queues consumer http add orders
curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" \
--header "Authorization: Bearer ${QUEUES_TOKEN}" \
--header "Content-Type: application/json" \
--data '{ "batch_size": 50, "visibility_timeout_ms": 60000 }'
curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/ack" \
--header "Authorization: Bearer ${QUEUES_TOKEN}" \
--header "Content-Type: application/json" \
--data '{ "acks": [{ "lease_id": "LEASE_1" }], "retries": [{ "lease_id": "LEASE_2", "delay_seconds": 60 }] }'go deeper
Know that a queue is consumed either by a Worker's queue() handler or by an HTTP client that pulls, never both.
Walk through enabling pull with wrangler queues consumer http add, the pull call's batch_size and visibility_timeout_ms, and acknowledging by lease_id.
Size the visibility timeout to processing time, back off on empty pulls, and keep producers off the v8 content type.
Weigh running consumers outside Workers, with tokens and capacity to manage, against push consumers that Cloudflare scales but that must fit Workers limits.
## Push and pull consumers **Cloudflare Queues** supports two kinds of consumer, and a queue has exactly **one type** configured at a time: - A **push-based consumer** is a Worker with a `queue(batch, env, ctx)` handler. Queues invokes it when messages are waiting, batches them by `max_batch_size` and `max_batch_timeout`, and scales concurrent invocations to keep up with the backlog. - A **pull-based consumer** is any program that speaks HTTP. It calls the Queues REST API to fetch a batch when it is ready, then reports which messages it finished. | Concern | Push (Worker) | Pull (HTTP) | |---|---|---| | Where the code runs | a Cloudflare Worker | anywhere with network access | | Who decides when to read | Queues | the client | | Batch shape | `max_batch_size`, `max_batch_timeout` | `batch_size` per pull | | Empty queue | no invocation, no cost | a pull still incurs a read operation | | Credentials | the binding, nothing to manage | an API token with Queues read and write | ## When pull is the better fit 1. **The processing code lives outside Workers**, for example existing services in another cloud or on premises, or code in a language or runtime the team does not run on Workers. 2. **The consumer must set its own pace**, because the work is long-running or the downstream resource, such as a pool of GPU machines, can only take so much at once. 3. **Several independent machines share the work.** Multiple HTTP clients can pull from one queue; each gets a different batch and holds the lease on it. The docs recommend starting with a push consumer, and the consumer type can be changed later by removing one and attaching the other. ## Enabling it Pull is enabled through Wrangler or the dashboard, not the Wrangler file. `wrangler queues consumer http add orders` attaches a pull consumer; if a Worker consumer is attached, remove it first with `wrangler queues consumer worker remove orders fulfilment` and delete its `queues.consumers` entry, or the next deploy will try to re-add a conflicting consumer. Wrangler rejects a `queues.consumers` entry whose `type` is not `worker`. The client needs an **API token** with both Queues read and Queues write permissions; acknowledging a message mutates the queue, so read-only is not enough. The token is sent as `Authorization: Bearer <token>` and should live in the client's secret store. ## Pulling The client POSTs JSON to `/accounts/{account_id}/queues/{queue_id}/messages/pull` with two optional fields: - `batch_size`: how many messages to return, **default 5, maximum 100**. - `visibility_timeout_ms`: how long the client holds a lease on the returned messages, **default 30 seconds, maximum 12 hours**. The response lists messages with `body`, `id`, `timestamp_ms`, `attempts`, `metadata` and a `lease_id`. Queues uses **short polling**: it answers immediately with what is available, or with an empty list, and never holds the connection open, so an idle client must back off between pulls to avoid paying for empty reads. ## Acknowledging The client POSTs to `/messages/ack` with an `acks` array and a `retries` array of `{ "lease_id": ... }` objects; a retry can add `delay_seconds`. A message neither acknowledged nor marked for retry becomes deliverable again when its visibility timeout expires, possibly to another client, so delivery stays **at least once**. Acknowledgements can be grouped to save API calls, and Queues still accepts an acknowledgement that arrives after the timeout. ## Operating a pull consumer - **Back off when a pull comes back empty.** Short polling means a tight loop on an idle queue pays a read for every empty answer. - **Watch the backlog in the response.** Each pull result includes `message_backlog_count`, which a fleet of clients can use to decide when to add or remove machines. - **Retry explicitly.** A message that failed should go in `retries`, optionally with `delay_seconds`, so it returns at once instead of after the lease expires. - **Batch the acknowledgements.** One `/messages/ack` call per pulled batch keeps API calls, and the client's own latency, down. - **Rotate the token like any other credential**, since it can read and remove messages from the queue. ## Content types A pull consumer reads bodies encoded inside JSON. It can handle `text`, `bytes` and `json` messages, decoding each by its content type, but it **cannot decode `v8`**, which is specific to the Workers runtime. Producers feeding a pull consumer should send one of the other three.
- Why does a Cloudflare Queues pull consumer's API token need write permission when it only reads messages?Consuming changes the queue's state: acknowledging a message removes it and marking one for retry puts it back. Both go through the `/messages/ack` endpoint, which needs Queues write. With read alone, the client could pull messages but never acknowledge them, so every message would reappear when its visibility timeout expired.
- How should a Cloudflare Queues pull consumer choose its visibility_timeout_ms?Set it above the longest time the client needs to process a batch, with margin. Too short and messages become deliverable again while still being worked on, so another client processes them too; too long and a crashed client's messages stay hidden until the lease ends. Marking a message for retry returns it at once instead of waiting.
- A queue can have a Worker consumer and a pull consumer at once.
- Set type to http_pull in the Wrangler file to enable pulling.
- A pull on an empty queue waits until a message arrives.
- A read-only API token is enough for a pull consumer.
- Pull consumers can decode v8 message bodies.