skip to content

With the GitHub CLI, how do gh api --paginate, --jq and --slurp interact when a script lists every repository in an organisation?

level: middleimportance: should knowfreq 26%

answer

  1. following the next link
  2. a hundred per page
  3. the filter runs per page
  4. one outer array, but no filter
  5. list commands stop at a limit

basics

~20 s

--paginate follows every next page (100 items each for REST); --jq then runs once per page, not over the whole set; --slurp wraps all pages into one array but cannot be combined with --jq, so whole-set filtering happens in an external jq.

solid answer

~40 s

`--paginate` keeps requesting until the `Link` header has no `next` page, and for REST it adds `per_page=100` unless you set it. `--jq` is evaluated **once per page**, so a stream expression like `.[].full_name` gives the full list, but an aggregate like `length` prints one number per page. `--slurp` (which requires `--paginate`) wraps every page into one outer JSON array, an array of page arrays for REST; it refuses to combine with `--jq` or `--template`, so totals and sorts happen by piping into a separately installed jq. gh's own list commands are the other route: `gh repo list acme --json nameWithOwner --jq ...` needs `--json` before `--jq` works and stops at `--limit`, which defaults to 30.

code

bash · 10 lines
bash
# One line per non-archived repository, across every page
gh api --paginate 'orgs/acme/repos?type=all' \
  --jq '.[] | select(.archived | not) | .name'

# One total for the whole organisation: slurp, then count in jq
gh api --paginate --slurp orgs/acme/repos | jq 'map(length) | add'

# The same list through gh's own command; raise the default limit of 30
gh repo list acme --no-archived --limit 1000 \
  --json nameWithOwner --jq '.[].nameWithOwner'

go deeper

for a junior

Remember that --paginate fetches every page and that --jq on list commands needs --json first.

for a middle

Explain that the jq filter runs per page, why counting and sorting break under --paginate, and what --slurp produces for REST pages.

for a senior

Show how you keep an org-wide report correct: streaming selections in --jq, whole-set work after --slurp, list-command limits raised, and request counts kept in mind.

for a principal

Judge when a nightly report built from paginated CLI calls should become a scheduled job using the GraphQL API or an app, given request budgets and maintainability.

## The scenario A platform engineer needs the name of every non-archived repository in the `acme` organisation, as one line each, to feed a nightly governance report. The organisation has 250 repositories. Three `gh api` flags decide whether the script sees all of them, and in what shape. ## `--paginate`: following the pages GitHub's REST list endpoints return one page at a time and advertise the next page in the `Link` response header. With `--paginate`, gh reads that header and keeps requesting until there is no `rel="next"` link. - For REST calls gh appends `per_page=100` to the path unless the path or the fields already set `per_page`, so 250 repositories take three requests. - `--paginate` only works for `GET` requests and for `graphql`; combining it with fields means passing `--method GET`, and it cannot be combined with `--input`. - Each page is a separate request against the API's rate limit; a walk over thousands of items is cheap, a walk that then makes one call per item is not. ## `--jq`: a filter that sees one page at a time `--jq` runs gh's built-in jq **on each page's response separately**. That is invisible for streaming filters and very visible for aggregates: | Filter | What you get for 250 repositories | |---|---| | `.[].full_name` | 250 lines, which is exactly what a script wants | | `.[] \| select(.archived \| not) \| .name` | Every non-archived name, one per line | | `length` | Three numbers: 100, 100 and 50 | | `sort_by(.pushed_at)` | Three separately sorted chunks, not one sorted list | The rule of thumb: inside `--paginate`, use `--jq` to **select and project**, never to count, sort, group or deduplicate. ## `--slurp`: one document for the whole set `--slurp` makes gh write every page into a single outer JSON array, in the manner of `jq --slurp`: 1. It requires `--paginate`; on its own gh refuses it. 2. It cannot be combined with `--jq` or `--template` (gh exits with the message that `--slurp` is not supported with them). 3. For REST the result is an **array of page arrays**, so a total is `map(length) | add` and a flat list is `add` or `[.[][]]` in an external jq. 4. For GraphQL, where each page is a separate response object, it gives one array of those objects. Without `--slurp`, gh still joins REST pages into one valid JSON array when its output goes to a pipe, so `gh api --paginate orgs/acme/repos | jq length` also works; `--slurp` matters most for GraphQL and for an explicit, predictable shape. ## gh's own list commands: `--json`, `--jq`, `--template` Many gh subcommands have structured output too. `gh repo list acme --json nameWithOwner,isArchived --jq '.[] | select(.isArchived | not) | .nameWithOwner'` gives the same list without any endpoint path. Two differences matter: - `--jq` and `--template` on these commands require `--json` with a field list; running `--json` with no fields prints the available names. - They fetch up to `--limit`, which is **30** for `gh repo list`, so an organisation-wide report needs `--limit 1000` or similar, while `gh api --paginate` simply walks to the end. The list command's filters, such as `--no-archived`, can replace part of the jq expression. ## Mistakes that make a report quietly wrong 1. **Counting inside `--jq`.** `--jq 'length'` under `--paginate` prints one number per page; a script that reads only the first line reports 100 for a 250-repository organisation. 2. **Sorting or deduplicating inside `--jq`.** Each page is sorted on its own, so the combined output is not sorted. 3. **Forgetting the list command's limit.** `gh repo list` returns 30 rows unless `--limit` is raised, with no warning that more exist. 4. **Adding fields without `--method GET`.** Fields turn the call into a `POST`, so the list endpoint rejects it instead of paging. 5. **Expecting `--slurp` to filter.** It only wraps pages; the filter has to run in a separately installed jq afterwards. Each of these produces output that looks plausible, which is why the nightly report should also print a total that someone can sanity-check against the organisation's repository count. ## Choosing - Use the list command when one exists and its fields cover the report. - Use `gh api --paginate --jq` for endpoints with no subcommand and for streaming selections. - Use `--paginate --slurp` piped into jq for anything that must see the whole set at once.

  • Why does gh api --paginate with --jq 'length' print several numbers?
    gh evaluates the jq filter on each page's response as it arrives, so `length` reports the size of every page (100, 100, 50 for 250 items). For a single total, drop `--jq`, add `--slurp` and run `map(length) | add` in an external jq.
  • When would you choose gh repo list --json over gh api orgs/acme/repos?
    When its JSON fields cover the report: the command knows its endpoint, offers filters such as `--no-archived` and names fields like `nameWithOwner`. Remember to raise `--limit`, which defaults to 30. Reach for `gh api` when the data has no subcommand.

  • --jq sees all pages merged when --paginate is used
  • --slurp and --jq can be combined to count every item
  • gh api --paginate uses the API's default page size of 30
  • gh repo list --json returns every repository in the organisation
  • --paginate works for POST and PATCH requests too