Common pitfalls and gotchas
This chapter collects the friction points that surfaced while exercising the live API during the writing of this guide. They are not bugs in the strict sense — many are documentation gaps, others are server-side behaviours that disagree with the docs — but each is the kind of thing that wastes a half-day of debugging the first time you hit it. Read this chapter once now; come back to it whenever something behaves unexpectedly. Items with a canonical home in another chapter are summarised here in one line and cross-referenced; items unique to this chapter carry their full text.
The top items
1. The X-Pagination-* response headers are content-type-conditional and partly empty.
The Response Headers tutorial promises three pagination
headers. What the dev environment actually emits:
X-Pagination-Limit and X-Pagination-Next-Offset
appear on response_format=csv and csv_gzip
responses but not on json; X-Pagination-Offset
appears on those same responses but comes back empty-valued on
page 1; the documented X-Pagination-Has-Next is not emitted
at all. So a client that reads the headers as the primary pagination
signal will both miss them entirely on JSON and read a blank
"current offset" on the first CSV page.
Recommendation: read
pagination.next_offset from the JSON body (terminates
when null). When pulling CSV/CSV-gzip in production,
either fetch the first page as JSON to read the pagination envelope
or trust X-Pagination-Next-Offset on the CSV response
(present but content-type-conditional). The body signal is the only
universally-available one.
2. The Link response header is emitted, but the URL is unfollowable.
The deployed dev API emits an RFC 5988
link: <URL>; rel="next" header on paginated JSON and CSV
responses. However the URL it carries is unusable: it points at an
internal origin host over http:// rather than at the public
https://dev-datasets-api.algoseek.com edge. Captured live:
link: <http://3.89.22.195/api/v1/data/us-equity/eq-trades/2023-01-17/AAPL?adjusted=true&offset=3&limit=3>; rel="next"
Following that URL would (a) bypass CloudFront and the
X-API-KEY edge check, (b) transmit your key over plain
HTTP, and (c) break the moment the upstream IP rotates. So for
practical purposes the header still cannot be used by clients.
Recommendation: drive your walk off
pagination.next_offset in the JSON body. Do not follow
the Link URL. Internally, the algoseek team should consider
either rewriting the URL at the edge to use the public hostname over
HTTPS, or suppressing the header on responses that traverse the
gateway.
3. Column-name filters are not in the OpenAPI spec.
The OpenAPI spec at /openapi.json lists only a few standard
parameters per data endpoint
(sort, columns, offset,
limit, response_format, plus a few endpoint-specific
flags). It does not list the column-name filters
(Ticker, TradeDate, etc.) that most data
endpoints accept — and that some require.
Recommendation: the column schema for a dataset is the
authoritative source for legal filters. Fetch
GET /api/v1/meta/datasets/{id}/columns once at startup,
cache the result, and validate user-supplied filter names against
the cached set before issuing any data call.
Opaque catalog id, not text id.
The {dataset_id} placeholder on the catalog endpoints
takes the opaque catalog id (a string of the form
US####), not the human-readable
dataset_text_id. See the catalog chapter for
the canonical statement of the rule.
4. URL paths and entitled dataset_text_ids do not always match 1:1.
The same URL path can route to different underlying datasets
depending on flag-style query parameters. Adding
adjusted=true routes eq-trades to
eq_trades_adj; adding
aggregation_logic=industry_std routes
eq-daily-ohlc to eq_daily_ohlc_ind_std. The
default URL with no flags maps to a specific
dataset_text_id that may not be in your subscription, even
when a variant is.
Recommendation: on a 403 from a data endpoint,
cross-reference the URL family against
/account/my/data-access-rules and try the routing flags
that match a granted variant.
5. Required-filter behaviour varies by endpoint.
Some query-style endpoints (e.g. fut-trades-1min,
eq-trades-1min) require at least one column-name filter
(typically Ticker) and return 422 with
"The filter expression for the field 'Ticker' should be specified"
if you omit it. Other query-style endpoints
(e.g. eq-shares-outst-basic, eq-market-holidays)
return rows without any filter.
Recommendation: until the spec lists per-endpoint required
filters, the safest pattern is to always pass at least one identifier
filter on data endpoints; reference endpoints (anything under
/us-equity-ref/) generally tolerate filterless calls.
6. pagination.next_offset signals end-of-walk via null.
Terminate the pagination loop on
pagination["next_offset"] is None — not on a non-existent
has_next boolean. Type the field as
Optional[int]. See the pagination chapter
for the full account.
7. end_date is null on ongoing datasets.
The GET api/v1/meta/datasets/{id}/info response gives
"end_date": null for any dataset still being updated.
The next paragraph of the description says "the dataset is updated
daily," so the meaning is "ongoing," not "broken."
Recommendation: treat end_date == null as
"ongoing," not as missing data. For UI displays, render it as
"ongoing" or "current."
8. aggregation_logic accepts only two values.
The legal values are algoseek and industry_std.
Other values you might guess from the granted dataset names
(ind_std, primary_adjusted, etc.) return
422 with a helpful enum-error envelope listing the legal
values.
Recommendation: read the enum error envelope on the first attempt; the message contains the canonical legal-values list.
9. The futures contract symbol uses single-digit year.
ESH3, not ESH23. The two-digit-year form returns
data:[] (200 OK with no data), which is harder to detect
than a 422.
Recommendation: when probing futures, always test with a
known good single-digit-year symbol (ESH3 for March 2023
E-mini S&P) before constructing your own.
10. Adjusted vs. unadjusted boundaries.
This is the most common silent-failure mode in market-data pipelines
generally, and the API exposes it through the adjusted=true
flag. Adjusted prices fold split and dividend factors into the price
itself; unadjusted prices preserve the raw level traded. The two
cannot be meaningfully compared point-to-point or aggregated, and
mixing them in joins is the single most common source of subtle
backtest bugs.
Recommendation: commit your pipeline to one or the other at
the schema level. Store an explicit is_adjusted flag on
every persisted row; assert on it during every load, every join,
every merge. For the pipelines that need both (e.g. simulating an
order at the historical raw price but reporting P&L on the adjusted
basis), keep the two universes in separate tables and only join at
the very last moment, with the adjustment factor explicitly visible
in the query plan.
11. Bad path-segment dates return 422, not 404.
A malformed date in the path segment (e.g. /eq-trades/9999-99-99/AAPL)
returns 422 Unprocessable Entity with body
"Cannot parse the filter value '9999-99-99' as Date", not
404. 404 is reserved for genuinely-missing
paths (/api/v1/foo/bar, see pitfall 13 below).
Recommendation: treat 422 as the canonical bad-date error
class and 404 as the canonical missing-resource class; they are
distinct on this API and a single except 404
catch-all in the client will misroute date-parse errors.
12. HEAD and OPTIONS are not what HTTP convention suggests.
HEAD /api/v1/status returns 405 Method Not Allowed
with allow: GET. OPTIONS /api/v1/status without
an Origin header also returns 405. With an
Origin header from an unrecognised origin, the API responds
400 with body "Disallowed CORS origin" but the
response also carries
access-control-allow-methods: DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT,
access-control-allow-credentials: true,
and access-control-max-age: 600 — so CORS preflight is
configured for a specific origin allow-list (not advertised
publicly), but HEAD requests are flat-out unsupported on
the data plane.
Recommendation: use GET only. Do not rely on
HEAD for liveness or content-length probes. If you need
CORS access from a browser context, contact your account contact
to register your origin.
Captured evidence.
The f32_head_405 probe in
smoke_tests/repro_F24_to_F33.py confirms 405 with
allow: GET on HEAD /api/v1/status; the same script
exercises the OPTIONS-without-Origin 405 and the
disallowed-origin 400. No standalone captured-headers file is checked
in for the 405 response.
13. Unknown paths return a clean 404 envelope.
GET /api/v1/foo/bar returns
{"detail":"Not Found"} with content-type: application/json.
The body shape matches the documented error
envelope, so a single error-handling code path covers both
path not found and filter rejected.
14. The X-API-KEY header tolerates surrounding whitespace and lowercase casing.
X-API-KEY: key (with leading or trailing whitespace inside
the value) returns 200 just like the strict form —
the API trims the value before comparison. x-api-key:
(lowercase header name) also works — HTTP/2 normalises header
casing on the wire. Missing the header returns 401 with
{"detail":"Not authenticated"}.
Recommendation: send X-API-KEY: <key> verbatim —
the Python requests client refuses to transmit a header
value that begins with whitespace — but you do not need to be
paranoid about casing.
Advanced gotchas summary
A handful of advanced gotchas have their full text in other chapters and were merely cross-referenced here; for the reader's convenience they are collected as one-line entries below.
| # | Topic | Canonical home |
|---|---|---|
| A1 | Quotas pool across keys; do not fan out by key. | quotas |
| A2 | Two request-id headers with different UUIDs on 2xx; only the AWS id is reliably present on 4xx/5xx. | errors |
| A3 | /account/my/quotas is itself counted toward the per-minute bucket; do not poll during cooldown. | quotas |
| A4 | Per-minute counter is leaky-bucket-shaped and counts refused 429 calls; sleep ~90 s on 429. | quotas |