Skip to main content

Pagination: walking large result sets

Most data endpoints in the API can return more rows than fit in a single response. The protocol gives you one body-driven way to walk the rest: an explicit limit/offset pair plus the pagination.next_offset field in the JSON envelope. A nominal link: rel="next" response header also appears on paginated responses, but is unfollowable on this deployment — see the gotcha below.

The two parameters that bound a page​

Every data endpoint accepts two query parameters that together define a page of results:

  • limit — The maximum number of records to return in this response. Defaults to 1000; the server will not return more than 10 000 even if you ask for it.
  • offset — The number of records to skip before the first row in this response. Defaults to 0.

Limit/offset is simple, debuggable, and stateless — the canonical choice for backfills and ad-hoc analysis. It is, however, prone to two well-known failure modes: at very deep offsets the engine still has to count past every preceding row (linear cost), and any concurrent insert/update upstream of your offset will shift rows under you (skipped or repeated rows). Neither matters for a static historical dataset queried with a stable sort, which is the common case here. For relentlessly streamed data you would want a cursor keyed off a monotonic field instead.

Reading where you are: the JSON body​

Every JSON response from a paginated endpoint embeds a pagination object alongside data:

Captured response body — page_eq_trades_p1
{
  "data": [
    {
      "TradeDate": "2023-01-17",
      "EventDateTime": "2023-01-17T10:03:08.220987-05:00",
      "EventType": "TRADE",
      "Ticker": "AAPL",
      "ASID": 1010000000001033,
      "Price": 133.696,
      "Quantity": 53,
      "Exchange": "NASDAQ",
      "ConditionCode": 2684354593
    },
    {
      "TradeDate": "2023-01-17",
      "EventDateTime": "2023-01-17T10:03:08.257651-05:00",
      "EventType": "TRADE",
      "Ticker": "AAPL",
      "ASID": 1010000000001033,
      "Price": 133.7044,
      "Quantity": 2,
      "Exchange": "FINRA",
      "ConditionCode": 2147483649
    },
    {
      "TradeDate": "2023-01-17",
      "EventDateTime": "2023-01-17T10:03:08.270364-05:00",
      "EventType": "TRADE",
      "Ticker": "AAPL",
      "ASID": 1010000000001033,
      "Price": 133.701,
      "Quantity": 1,
      "Exchange": "FINRA",
      "ConditionCode": 2147483649
    }
  ],
  "pagination": {
    "offset": 0,
    "limit": 3,
    "next_offset": 3
  }
}

The fields are:

  • pagination.offset — The offset of the current page (echoes the value the server interpreted from your request).
  • pagination.limit — Echoes the limit you asked for.
  • pagination.next_offset — An integer giving the offset to use for the next page, or null when the current page is the last one. This is the canonical end-of-walk signal.

:::caution Gotcha The pagination object on the deployed API is {offset, limit, next_offset}; there is no has_next boolean. Terminate the walk on pagination["next_offset"] is None (Python) / pagination.next_offset === null (JS); a typed client should declare next_offset as Optional[int] / int | null / *int. :::

:::caution Gotcha On responses where data is empty, pagination.offset comes back as null rather than as an integer. If you have declared offset as int in a typed client (Pydantic, TypeScript, Go), use Optional[int] / int | undefined / *int so empty days do not crash the parser. :::

The same pagination state is nominally also exposed as an RFC 5988 link: <URL>; rel="next" HTTP response header. Inspect the response headers and you will see something like:

Response headers — page_eq_trades_p1
Content-Type: application/json
Content-Length: 704
Connection: keep-alive
Date: Fri, 08 May 2026 20:18:39 GMT
x-amzn-Remapped-Date: Fri, 08 May 2026 20:18:39 GMT
x-amzn-RequestId: f248981c-076d-4282-a00a-eaa2870ba925
x-amzn-Remapped-Content-Length: 704
x-amzn-Remapped-Connection: keep-alive
x-request-id: 9b8fcf85-cb5b-4aae-b1f7-d1639c41f9c1
x-amz-apigw-id: dD-9CHfKIAMEnkQ=
x-amzn-Remapped-Server: nginx/1.18.0 (Ubuntu)
link: <http://3.89.22.195/api/v1/data/us-equity/eq-trades/2023-01-17/AAPL?adjusted=true&offset=3&limit=3>; rel="next"
X-Cache: Miss from cloudfront
Via: 1.1 1223fdba7e12210962498f19cd8ed6d8.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: GRU3-P3
X-Amz-Cf-Id: eK5LLQK_Wq6bEo4UAThDCxZRY7kH827y46clDj7JZEqfhKQy5Dlr2A==

Do not follow the Link URL. On the deployed dev API the URL it carries points at an internal origin host over plain http:// and is unsafe to follow; the body's pagination object is the only correct pagination signal. See pitfalls item 2 for the full account.

The X-Pagination-Limit, X-Pagination-Next-Offset, and X-Pagination-Offset response headers are emitted on csv and csv_gzip responses but not on json; on page 1 the X-Pagination-Offset value comes back empty. The canonical signal remains the body's pagination.next_offset — see pitfalls item 1.

Walking a paginated dataset (Python)​

A production-grade walker should:

  1. carry the per-call timing for telemetry,
  2. respect a minute-quota by sleeping when the quota header / GET api/v1/account/my/quotas indicates the next request would exceed the limit,
  3. be re-entrant: if the loop is interrupted, it should resume from the last successful offset rather than restart,
  4. validate the response against the expected schema, and
  5. retry transient errors (5xx, network timeouts) with capped exponential backoff and jitter.

The full implementation appears in the ingestion-client chapter. Below is the kernel of the loop, decoupled from the production cross-cutting concerns:

Pagination kernel. Yields one record at a time so the caller can stream-process without buffering the whole result set in memory — important for full TAQ days that exceed RAM. The caller's session must have X-API-KEY set per the auth chapter.
from __future__ import annotations
from typing import Iterator
import requests

def paginate(session: requests.Session, url: str,
             params: dict, page_size: int = 10_000) -> Iterator[dict]:
    next_offset: int | None = 0
    while next_offset is not None:
        page_params = {**params, "limit": page_size, "offset": next_offset,
                       "response_format": "json"}
        resp = session.get(url, params=page_params, timeout=60)
        resp.raise_for_status()
        payload = resp.json()
        for row in payload.get("data", []):
            yield row
        next_offset = (payload.get("pagination") or {}).get("next_offset")

Choosing a sensible page size​

The default page size is 1 000 records; the maximum is 10 000. The trade-off is between per-request overhead and per-request risk:

  • Smaller pages (e.g. 100–500) make individual requests faster, fail less catastrophically (a transient timeout costs you only that page, not 10 000 records you have to refetch), and stream nicely into memory-bounded consumers.
  • Larger pages (e.g. 5 000–10 000) reduce the per-request fixed overhead — the latency of TLS, routing, entitlement check, and JSON serialisation — which dominates on small responses.

For large historical backfills that need to complete in bounded wall-clock time, 10 000 minimises the per-request fixed cost. For streaming-style consumers that want to start emitting downstream rows as soon as possible, 1 000 strikes a better balance. Always benchmark against your own pipeline; the answer changes with response width (number of columns) more than with response height (number of rows).

Verifying you walked the whole dataset​

A simple post-condition for any pagination walk is: did the last response have pagination.next_offset == null? If yes, you saw every row. If no (e.g. your loop exited early because of an exception), you stopped partway and need to resume.

Backfill checkpointing.​

Persist the (URL, query, last successful offset) tuple after every page. On restart, resume from the checkpoint rather than from offset 0. For idempotency, identify the "natural key" of each row (typically TradeDate + EventDateTime + Ticker + Sequence or similar) and use a target-store upsert so re-fetching the last page on resume is harmless. The ingestion-client chapter walks through the full implementation.