How to Paginate Large API Datasets: Limit & Offset

When working with high-resolution financial market data, response payloads can become exceptionally large. The algoseek Datasets API provides a robust set of query parameters designed to help you efficiently paginate through millions of rows, optimize network transfer sizes, and shape the data to fit your exact analytical models.

This guide details the standard query parameters available across the /data/ endpoints and demonstrates how to implement them effectively.

Response Formatting (response_format)

Because the pagination method you use depends on the selected response format, you should decide your format first. While JSON is the default format, it is not always the most efficient choice for bulk ingestion into pandas DataFrames, relational databases, or quantitative frameworks. The API supports direct CSV and compressed CSV downloads.

Options:

• json (Default)
• csv (Returns text/csv data)
• csv_gzip (Returns application/gzip containing data in CSV format)

Example Request (Download compressed CSV):

curl --get \
    'https://api.algoseek.com/v1/data/us-equity/taq/2023-08-02/AAPL' \
    --data-urlencode 'response_format=csv_gzip' \
     -H "X-API-KEY: YOUR_API_KEY" \
    --output 'AAPL_20230802_TAQ.csv.gz'

Pagination with JSON response

Data endpoints return results in subsets (pages) to ensure system stability and predictable response times. You control pagination using the limit and offset parameters.

• limit: Defines the maximum number of records to return in a single request.

• Default: 1000

• Maximum: 10000

• offset: Specifies the number of records to skip before beginning to return data.

• Default: 0

Note on query page limits

Here are the query limits for the different output formats:

• JSON: 10,000
• CSV: 30,000
• CSV Gzip: 80,000

Handling the Pagination Response: When returning JSON, the API includes a pagination metadata block alongside your data array. You should rely on the next_offset value to determine if you need to make subsequent requests.

Example Request:

curl --get \
    'https://api.algoseek.com/v1/data/us-equity/daily-ohlc/AAPL' \
     --data-urlencode 'limit=1000' \
     --data-urlencode 'offset=0' \
     -H "X-API-KEY: YOUR_API_KEY"

Example Response Block:

{
    "data": [
        {
            "TradeDate": "2007-01-03",
            "Ticker": "AAPL",
            "ASID": 1010000000001033,
            "OpenPrice": 86.28,
            "HighPrice": 86.58,
            "LowPrice": 81.9,
            "ClosePrice": 83.76,
            "MarketHoursVolume": 43432630,
            "MarketHoursFinraVolume": 10696551,
            "DailyVolume": 45105870,
            "DailyFinraVolume": 11540291,
            "MarketHoursVWAP": 84.848,
            "DailyVWAP": 84.8501
        },
        {
            "TradeDate": "2007-01-04",
            "Ticker": "AAPL",
            "ASID": 1010000000001033,
            "OpenPrice": 84.17,
            "HighPrice": 85.95,
            "LowPrice": 83.82,
            "ClosePrice": 85.66,
            "MarketHoursVolume": 29812464,
            "MarketHoursFinraVolume": 7804137,
            "DailyVolume": 30928848,
            "DailyFinraVolume": 8317690,
            "MarketHoursVWAP": 85.1213,
            "DailyVWAP": 85.1256
        },
        {
            "TradeDate": "2007-01-05",
            "Ticker": "AAPL",
            "ASID": 1010000000001033,
            "OpenPrice": 85.84,
            "HighPrice": 86.2,
            "LowPrice": 84.4,
            "ClosePrice": 85.05,
            "MarketHoursVolume": 29443139,
            "MarketHoursFinraVolume": 6892481,
            "DailyVolume": 30202106,
            "DailyFinraVolume": 7133007,
            "MarketHoursVWAP": 85.1939,
            "DailyVWAP": 85.1969
        },
        {
            "TradeDate": "2007-01-08",
            "Ticker": "AAPL",
            "ASID": 1010000000001033,
            "OpenPrice": 85.98,
            "HighPrice": 86.53,
            "LowPrice": 85.28,
            "ClosePrice": 85.47,
            "MarketHoursVolume": 28103154,
            "MarketHoursFinraVolume": 7747266,
            "DailyVolume": 28701137,
            "DailyFinraVolume": 7896292,
            "MarketHoursVWAP": 85.9176,
            "DailyVWAP": 85.9128
        },
        ...
    ],
    "pagination": {
        "offset": 0,
        "limit": 1000,
        "next_offset": 1000
    }
}

Requesting the Next Page: To fetch the next page, extract the next_offset value from the response block and execute the same query with offset set to that value. If next_offset returns null, you have reached the end of the dataset.

curl --get \
    'https://api.algoseek.com/v1/data/us-equity/daily-ohlc/AAPL' \
     --data-urlencode 'limit=1000' \
     --data-urlencode 'offset=1000' \
     -H "X-API-KEY: YOUR_API_KEY"

Pagination with CSV response

When requesting data using csv or csv_gzip formats, the response body contains raw tabular data rather than a structured JSON object. Because there is no JSON metadata block to hold the pagination state, the method for retrieving the next page differs.

Using Pagination Response Headers: For CSV responses, you must monitor the HTTP response headers returned by the API. Look for the API's custom pagination headers (e.g., X-Pagination-Next-Offset) to determine if there is more data to fetch and what your next offset should be.

CSV Header Row Logic: To make merging multiple paginated CSV files seamless, the API dynamically handles the output of the CSV column names (the header row):

• The CSV file header row is only included on the first page (when offset=0 or is omitted).
• For all subsequent pages (e.g., offset=5000), the API returns only the data rows. This allows you to safely append the raw CSV text from paginated responses directly to an existing file without having to manually strip out duplicate headers.