Rate limits and quotas: scheduling, backoff, and quota arithmetic

The API enforces three independent quotas, each tracked at two horizons (per minute and per month):

requests — The number of API calls. The simplest bucket; easy to reason about.
data_scanned — The number of bytes the engine reads to assemble your responses. Dominated by the underlying scan cost — large for unfiltered or wide queries, small for narrow filtered queries.
data_returned — The number of bytes actually sent over the wire to you. Always smaller than data_scanned; response_format=csv_gzip reduces it further.

The three quota dimensions are not interchangeable: a high data_scanned workload that returns very little data (e.g. a COUNT-style aggregation) will trip the scan quota long before either of the other two; a tight loop of small one-row queries will trip the requests quota long before either byte-count quota. Design your client to know which dimension is the binding constraint and to plan its calls against it.

Reading current usage

The GET /api/v1/account/my/quotas endpoint returns both the current usage and the configured limits, both at the per-month and per-minute horizons:

Captured response body — iam_account_quotas
{
  "quotas_usage": {
    "month": {
      "requests": 546,
      "data_scanned": 3139447119,
      "data_returned": 842277070
    },
    "minute": {
      "requests": 1,
      "data_scanned": 502,
      "data_returned": 256
    }
  },
  "quotas_limit": {
    "month": {
      "requests": 1000,
      "data_scanned": 100000000000,
      "data_returned": 10000000000
    },
    "minute": {
      "requests": 10,
      "data_scanned": 10000000000,
      "data_returned": 1000000000
    }
  }
}

The semantics:

The month bucket resets at 00:00 UTC on the first day of each month.
The minute bucket is a rolling 60-second sliding window, not aligned to wall-clock minute boundaries. Confirmed by recovery-timing probes: a 429 fired at $t_0$ remains in force until enough requests in the preceding 60 s have aged out for the count to drop back at or below the cap (typically ~60–70 s after the burst that tripped the limit; failed 429 attempts also count toward the bucket, so tight retry loops slow the recovery — see the gotchas below).
Quotas are pooled at the account level: every API key on the account contributes to the same buckets. Issuing more keys does not give you more headroom.

The 429 response

When you exceed the per-minute requests quota, subsequent calls return 429 Too Many Requests until enough time has passed for the rolling window to release a slot. Per-month overflow does the same but with a longer recovery time — you will need to wait until the next month, or contact your account contact to arrange a quota bump.

How the per-minute window actually behaves

The deployed API does not send a Retry-After header on a 429. The headers it does send are x-ratelimit-limit: 10 and x-ratelimit-remaining: 0; there is no x-ratelimit-reset. The body is a single detail string of the form "Per-minute request limit exceeded: N/10 requests.". The observed behaviour of the bucket has five distinct properties that all matter:

The window is rolling 60 s, not wall-clock minute. A 429 fired at $t_0$ remains in force until enough of the preceding 60 s of requests have aged out for the count to drop back at or below the cap.
Refused 429 attempts also count toward the bucket. Tight retry loops slow the recovery, not speed it up; the body's N/10 routinely reports N far larger than 10 (40/10 observed after only six fresh attempts following a 60-second cooldown).
GET /api/v1/account/my/quotas is itself counted toward the per-minute bucket. A sequence of /account/my calls during a 429 cooldown stays 200, but /account/my/quotas returns 429 and the refused poll itself counts toward the bucket — so polling /account/my/quotas as a backoff probe extends the lockout rather than clearing it.
quotas_usage.minute.requests lags actual usage by ~8–10 s under light load, longer under burst load. It is not safe to use as a real-time gate; treat it as a coarse monthly-bucket gauge and boot-time per-minute capacity hint.
Therefore: sleep ~90 s on 429, not 60 s, and do not poll GET /api/v1/account/my/quotas during the cooldown. 60 s is too short — a tight retry will re-trip the bucket because the leaky counter has not fully drained and your own refused 429 attempts kept it inflated. 90 s is the smallest value that consistently clears in our recovery-timing probes.

Build your own local rolling-window counter (a deque of monotonic timestamps; pop entries older than 60 s; admit when len(deque) < cap) and use it as the live gate; the token-bucket sketch later in this section is exactly that.

note

Endpoint exemptions. An endpoint-by-endpoint probe confirmed that the following endpoints are exempt from the per-minute bucket — they stay 200 regardless of poll rate, and burst calls against them did not increment quotas_usage.minute.requests on the next read: GET /api/v1/status, GET /api/v1/account/my, GET /api/v1/account/my/data-access-rules, GET /api/v1/meta/datasets/my, GET /api/v1/meta/datasets/{id}/info, and GET /api/v1/meta/datasets/{id}/columns. The GET /api/v1/meta/datasets catalog list is not exempt (see point 3 above). All GET /api/v1/data/... endpoints count against the per-minute bucket as expected.

A more efficient pattern is a local token-bucket simulator: read the per-minute capacity once at startup from GET api/v1/account/my/quotas, then meter your own request rate against a sliding minute window locally (the bucket below). The server's quotas_usage field lags actual usage by tens of seconds, so do not use it as the live gate — only as a boot-time capacity hint.

Token-bucket scheduler. Reads server limits once, then meters local request rate against a sliding minute window. Calls quotas endpoint to re-sync if the local view drifts (e.g. due to other processes on the same account).
from __future__ import annotations
import time
from collections import deque
from threading import Lock

class MinuteBucket:
    """Throttles to <= max_per_minute requests, sliding window."""
    def __init__(self, max_per_minute: int):
        self.cap = max_per_minute
        self.events: deque[float] = deque()
        self.lock = Lock()

    def acquire(self) -> None:
        while True:
            now = time.monotonic()
            with self.lock:
                # purge events older than 60s
                while self.events and now - self.events[0] >= 60.0:
                    self.events.popleft()
                if len(self.events) < self.cap:
                    self.events.append(now)
                    return
                # wait until the oldest event ages out; the loop then
                # rechecks under the lock. Not a busy-wait.
                wait = 60.0 - (now - self.events[0]) + 0.05
            # release lock before sleeping so other threads can drain
            time.sleep(max(wait, 0.05))

The bucket can also be initialised from the server's reported limits:

Bootstrap the local bucket from the server's quota response. Re-sync periodically.
from __future__ import annotations
import requests

def bootstrap_bucket(session, base_url):
    resp = session.get(f"{base_url}/api/v1/account/my/quotas", timeout=30)
    resp.raise_for_status()
    cap = int(resp.json()["quotas_limit"]["minute"]["requests"])
    return MinuteBucket(max_per_minute=cap)

Reducing scan cost (`data_scanned`)

data_scanned is the dominant cost on full-tick datasets and the easiest to overshoot accidentally. Three habits keep it bounded:

Always pass TradeDate. Most TAQ-style datasets are partitioned by trade date; a query without a date filter forces the engine to scan every partition.
Always pass an identifier filter (Ticker for equities, Ticker or BaseSymbol for futures). Without it, the engine returns the entire universe for the requested partition.
Don't ask for what you don't need. The columns projection does not reduce data_scanned (the engine still has to read the underlying columns), but it reduces data_returned which is the bandwidth quota.

Fan-out vs. bulk pull arithmetic.

For a backfill of $N$ ticker-days, you can either fan out one request per ticker-day (lots of small scans) or pull one ticker for many days at once (a few larger scans), depending on the dataset's partition layout. For trade-date partitioned datasets, the fan-out pattern is cheaper per row but more expensive in the requests bucket. Pick based on which quota dimension is your bottleneck; for accounts where the requests bucket is generous, fan-out wins on data_scanned efficiency.

Reducing wire cost (`data_returned`)

This is the easier optimisation: project to the columns you need (columns=) and pick response_format=csv_gzip when the result set is more than a few KB.

The relative compression ratio of gzip-CSV on TAQ data is roughly 8–10× (~87–90% reduction in data_returned); on highly repetitive reference data the observed ratio rises to 20× or more. The CPU cost on the receiving side is a few milliseconds per MB of compressed payload — effectively free unless you are running on a constrained edge device.

A simple quota-aware request wrapper

Quota-aware GET wrapper. Reads per-minute capacity once from /account/my/quotas, then meters all subsequent calls against a local sliding-window bucket. Falls back to a fixed ~90 s sleep on a 429 (no Retry-After is sent; 60 s can re-trip because refused 429s also count toward the bucket).
from __future__ import annotations
import time
from collections import deque
from threading import Lock
import requests

class QuotaAwareClient:
    def __init__(self, api_key: str, base_url: str = "https://dev-datasets-api.algoseek.com"):
        self.s = requests.Session()
        self.s.headers["X-API-KEY"] = api_key
        self.base = base_url
        # Bootstrap per-minute capacity from the quotas endpoint.
        # Called exactly once at startup; never during a backoff window
        # (see the rolling-window subsection above for why).
        r = self.s.get(f"{self.base}/api/v1/account/my/quotas", timeout=30)
        r.raise_for_status()
        self.minute_cap = int(r.json()["quotas_limit"]["minute"]["requests"])
        # Local sliding-window bucket. Authoritative; the server's
        # quotas_usage.minute.requests lags real usage by tens of seconds
        # and cannot be used as a live gate.
        self._events: deque[float] = deque()
        self._lock = Lock()

    def _admit(self) -> None:
        while True:
            now = time.monotonic()
            with self._lock:
                while self._events and now - self._events[0] >= 60.0:
                    self._events.popleft()
                if len(self._events) < self.minute_cap:
                    self._events.append(now)
                    return
                # wait until the oldest event ages out; not a busy-wait.
                wait = 60.0 - (now - self._events[0]) + 0.05
            time.sleep(max(wait, 0.05))

    # MAX_RETRIES = 6 here matches the production reference client in
    # the ingestion chapter. Three was the previous default; six is
    # sized for full-day backfills where a single transient 5xx burst
    # should not abort the pipeline.
    MAX_RETRIES = 6

    def get(self, path: str, *, params: dict | None = None) -> requests.Response:
        # Retry envelope: a server-side 429 (possible at the local-window
        # boundary because the server's rolling window can drift a few
        # seconds relative to our local clock) is absorbed here. Fixed
        # ~90s sleep on 429; see the rolling-window subsection above.
        delay = 1.0
        for attempt in range(self.MAX_RETRIES):
            self._admit()
            resp = self.s.get(f"{self.base}{path}", params=params, timeout=60)
            if resp.status_code < 400:
                return resp
            if resp.status_code == 429:
                # Body is a string detail of the form
                # "Per-minute request limit exceeded: N/10 requests."
                time.sleep(90.0)
                continue
            if 500 <= resp.status_code < 600:
                # Transient server-side failure: capped exponential backoff.
                if attempt == self.MAX_RETRIES - 1:
                    resp.raise_for_status()
                time.sleep(delay)
                delay = min(delay * 2, 30.0)
                continue
            # 4xx other than 429: permanent. Surface to caller.
            return resp
        # Retries exhausted on persistent 429 / 5xx: raise so callers
        # get a real exception rather than a silently-failing response.
        resp.raise_for_status()

The pattern above does not round-trip to GET api/v1/account/my/quotas on every call. The local sliding-window bucket is authoritative for per-minute pacing; the server-side counter lags real usage and GET api/v1/account/my/quotas is itself counted against the per-minute bucket (see the rolling-window subsection above for the full explanation and the endpoint-exemption list).

Reading current usage​

The 429 response​

How the per-minute window actually behaves​

Reducing scan cost (data_scanned)​

Fan-out vs. bulk pull arithmetic.​

Reducing wire cost (data_returned)​

A simple quota-aware request wrapper​