Pagination

The API implements pagination. By default, the page size is set based on your subscription. You can change the size of the page with the page_size parameter, up to the maximum allowed by your subscription. For instance, to access the second page where each page has 100 rows and you are using offset based pagination, append ?page=2&page_size=100 to your request URL.

Keep in mind that retrieving many records from large datasets might necessitate multiple requests. See below for information on how to determine if there is a next page.

Offset vs Cursor Pagination

• The API supports both offset and cursor pagination.

• Cursor pagination is used when the cursor parameter is provided.

• Cursor pagination is more efficient than offset pagination for large datasets because it doesn't require the API to scan through all the records to find the next page.

• For cursor pagination, for the first request, append cursor="" to your request to get the first page. For subsequent requests, use the cursor value from the meta data of the previous response.

• Offset pagination requires sending the page parameter to specify the page number.

JSON Response Format

When using the JSON response format, the API will return a meta object in the response that includes information about the current page, the limit, whether there are more records available (hasNextPage), and the cursor for using cursor pagination.

{
  "status": "success",
  "data": [...],
  "meta": {
    "page": 1,
    "page_size": 50000,
    "limit": 1000,
    "hasNextPage": true
    "cursor": "dGhpcyBpcyBhIGxvbmcgc3RyaW5n"
  }
}

Streaming the JSON Response

JSON responses are streamed: the API sends rows directly from the database as they are read, instead of buffering the entire result set in memory first, which lowers time-to-first-byte and memory use. The response body is identical to the non-streaming JSON format, including the meta object.

In streaming mode the pagination response headers (such as X-Has-Next-Page) are not sent, because the response headers are committed before all rows are read. Read pagination from the JSON meta.hasNextPage and meta.cursor fields. Streaming applies to the JSON response format only.

CSV Response Format

When using the CSV response format, the API will return the pagination information as headers in the response:

• X-Page: The current page number

• X-Page-Size: The page size

• X-Limit: The maximum number of rows to return across all pages

• X-Has-Next-Page: A boolean value indicating whether there are more records available