Skip to main content

Response formats: JSON, CSV, and streaming gzip

Every data endpoint can return its rows in one of three shapes, chosen via the response_format query parameter:

  • json (default) — The familiar {"data": [...], "pagination": {...}} envelope. Best for small slices and for responses you intend to walk record-by-record in a script.
  • csv — Plain comma-separated text. Best for medium-sized slices that go straight to pandas, R, or a spreadsheet.
  • csv_gzip — CSV compressed with gzip. Best for large result sets that travel over a slow network or that you want to write to disk untouched.

The wire-format choice is also a cost choice: data_returned in the quota object is byte-counted on the actual bytes sent, which means gzip-compressed responses consume meaningfully less of your monthly bandwidth quota than uncompressed equivalents. On TAQ-shaped payloads, empirically observed compression ratio is roughly 8–10× (~87–90% reduction in data_returned); see the quotas chapter.

Same data, three formats: a side-by-side comparison​

To make the differences concrete, here is the same query — one day of adjusted AAPL trades, limit 5 — in each format. Watch the response size in the per-call meta lines.

JSON​

JSON: the default; explicit here for clarity.
curl -s "https://dev-datasets-api.algoseek.com/api/v1/data/us-equity/eq-trades/2023-01-17/AAPL?adjusted=true&limit=5&response_format=json" \
     -H "X-API-KEY: $ALGOSEEK_API_KEY"
Captured response body — fmt_eq_trades_json
{
  "data": [
    {
      "TradeDate": "2023-01-17",
      "EventDateTime": "2023-01-17T12:35:55.383637-05:00",
      "EventType": "TRADE NB",
      "Ticker": "AAPL",
      "ASID": 1010000000001033,
      "Price": 133.7059,
      "Quantity": 100,
      "Exchange": "EDGX",
      "ConditionCode": 536870945
    },
    {
      "TradeDate": "2023-01-17",
      "EventDateTime": "2023-01-17T12:35:55.383868-05:00",
      "EventType": "TRADE NB",
      "Ticker": "AAPL",
      "ASID": 1010000000001033,
      "Price": 133.7059,
      "Quantity": 100,
      "Exchange": "NYSE",
      "ConditionCode": 536870945
    },
    {
      "TradeDate": "2023-01-17",
      "EventDateTime": "2023-01-17T12:35:55.397113-05:00",
      "EventType": "TRADE",
      "Ticker": "AAPL",
      "ASID": 1010000000001033,
      "Price": 133.6986,
      "Quantity": 27,
      "Exchange": "FINRA",
      "ConditionCode": 2147483649
    }
  ],
  "pagination": {
    "offset": 0,
    "limit": 5,
    "next_offset": 5
  }
}

CSV​

CSV: text/csv body. Note the absence of the JSON envelope — no data or pagination wrapper. CSV gives you the rows but no in-band pagination hint; see the caveat below.
curl -s "https://dev-datasets-api.algoseek.com/api/v1/data/us-equity/eq-trades/2023-01-17/AAPL?adjusted=true&limit=5&response_format=csv" \
     -H "X-API-KEY: $ALGOSEEK_API_KEY"
CSV response body — fmt_eq_trades_csv
"TradeDate","EventDateTime","EventType","Ticker","ASID","Price","Quantity","Exchange","ConditionCode"
"2023-01-17","2023-01-17 04:00:00.005848448","TRADE","AAPL",1010000000001033,132.4067,5,"EDGX",2147484673
"2023-01-17","2023-01-17 04:00:00.006254542","TRADE","AAPL",1010000000001033,132.4067,20,"EDGX",2147484673
"2023-01-17","2023-01-17 04:00:00.007442373","TRADE","AAPL",1010000000001033,132.4067,25,"EDGX",2147484673
"2023-01-17","2023-01-17 04:00:00.007456537","TRADE","AAPL",1010000000001033,132.4067,15,"EDGX",2147484673
"2023-01-17","2023-01-17 04:00:00.007463418","TRADE","AAPL",1010000000001033,132.4067,100,"EDGX",1025

Gzip-compressed CSV​

CSV gzip: binary body. Save it to a file rather than printing it. The Content-Disposition header tells you the suggested filename.
curl -s -O -J \
  "https://dev-datasets-api.algoseek.com/api/v1/data/us-equity/eq-trades/2023-01-17/AAPL?adjusted=true&limit=5&response_format=csv_gzip" \
  -H "X-API-KEY: $ALGOSEEK_API_KEY"
Response headers — fmt_eq_trades_csv_gzip
Content-Type: application/gzip
Content-Length: 228
Connection: keep-alive
Date: Fri, 08 May 2026 20:18:31 GMT
x-pagination-limit: 5
x-amzn-RequestId: f6a8bc92-7768-4515-9c00-2e592b8f8e21
content-disposition: attachment; filename=USEquityMarketData.TradeOnlyAdjusted.csv.gz
x-amzn-Remapped-Connection: keep-alive
x-request-id: 5924b865-9158-4479-a825-d5113b54fde3
x-pagination-next-offset: 5
x-pagination-offset:
x-amz-apigw-id: dD-7sGFMIAMEVhQ=
x-amzn-Remapped-Server: nginx/1.18.0 (Ubuntu)
link: <http://3.89.22.195/api/v1/data/us-equity/eq-trades/2023-01-17/AAPL?adjusted=true&response_format=csv_gzip&offset=5&limit=5>; rel="next"

For programmatic consumers, parse the Content-Disposition header rather than constructing the filename yourself — the server encodes the dataset variant, identifier, and date in a stable convention you can rely on for downstream partitioning.

Reading the gzip body without writing it to disk​

For full-day TAQ pulls (tens to hundreds of MB compressed), avoid loading the entire compressed body into memory. Stream it through requests.get(stream=True) and feed the chunked stream into gzip.GzipFile, which itself yields bytes that pandas can consume incrementally:

Streaming download + decompress + parse pipeline. Memory footprint stays bounded by the chunk size, not the file size. Replace process with your downstream consumer (DataFrame append, parquet writer, database upsert, etc.).
import os
import gzip
import pandas as pd
import requests

API_KEY = os.environ["ALGOSEEK_API_KEY"]
URL = ("https://dev-datasets-api.algoseek.com"
       "/api/v1/data/us-equity/eq-trades/2023-01-17/AAPL")

def process(chunk: pd.DataFrame) -> None:
    """Replace this with your downstream consumer."""
    print(f"got {len(chunk)} rows")

with requests.get(URL,
                  headers={"X-API-KEY": API_KEY},
                  params={"adjusted": "true", "limit": 10_000,
                          "response_format": "csv_gzip"},
                  stream=True, timeout=300) as resp:
    resp.raise_for_status()
    # the body is gzip-typed, not gzip-encoded -- do not set Accept-Encoding
    with gzip.GzipFile(fileobj=resp.raw) as gz:
        for chunk in pd.read_csv(gz, chunksize=10_000):
            process(chunk)

The Content-Disposition header is preserved on the response object (resp.headers["content-disposition"]). Parse it with the standard library helpers if you need the suggested filename for downstream partitioning.

When to pick which format​

Use casePick
Quick exploration in a notebookjson — easiest to inspect with print(json.dumps(...))
Loading a few thousand rows into pandascsv — pandas reads CSV faster than JSON, and there is no envelope to peel off
Backfilling a full historical day or yearcsv_gzip — the bandwidth savings dominate
Stream-processing without bounded memorycsv_gzip with stream=True
Pagination across many pagesjson for the first page to read pagination.next_offset, then csv_gzip for the subsequent pages once next_offset is known
Saving raw to a partitioned data lakecsv_gzip — store as-is, no re-encoding

The pagination caveat for CSV.​

Because CSV responses do not embed the JSON envelope, you cannot read pagination.next_offset from the body. The Link rel="next" header is also unsafe to follow on this API (see pitfalls item 2). The robust pattern is therefore: issue the first call as json to read pagination.next_offset (and to discover whether you need pages 2+), then issue subsequent pages as csv_gzip with the offsets you computed from the JSON walk for the bandwidth saving.

Why not Accept-Encoding: gzip?​

HTTP transport-level content negotiation is not honoured on the deployed API — neither for CSV nor for JSON responses. Setting Accept-Encoding: gzip on the request returns the body uncompressed: Content-Type: text/csv or application/json, no Content-Encoding: gzip, full plaintext on the wire. To get a compressed body you must opt in explicitly with response_format=csv_gzip, which returns Content-Type: application/gzip and a body whose first two bytes are the gzip magic 1f 8b. There is no equivalent explicit-opt-in for JSON — if you want compressed JSON the only option is to convert to CSV first. Treat response_format=csv_gzip as the only path to compressed data; do not rely on the transport layer to negotiate compression for you.

:::caution Gotcha The body served under response_format=csv_gzip is binary-safe gzip. Verify with the magic-byte check resp.content[:2] == b"\x1f\x8b" before piping into a gzip decoder, so a misrouted plain-CSV body fails loudly rather than corrupting the consumer. :::