Best Practices

Optimize your Grid Status API usage for performance, reliability, and cost-effectiveness.

Query Optimization

1

Always Use Time Filters

Unbounded queries can be slow and expensive. Always specify start_time and end_time when querying time-series data.

Bad

# Queries entire dataset - slow and expensive
curl -H "Accept-Encoding: br, gzip" "https://api.gridstatus.io/v1/datasets/ercot_lmp_by_settlement_point/query?api_key=YOUR_API_KEY"

Good

# Bounded time range with location filter - fast and efficient
curl -H "Accept-Encoding: br, gzip" "https://api.gridstatus.io/v1/datasets/ercot_lmp_by_settlement_point/query?\
start_time=2026-01-01&\
end_time=2026-01-02&\
filter_column=location&\
filter_value=HB_HOUSTON&\
api_key=YOUR_API_KEY"

2

Filter by Subseries

Most LMP and load datasets contain data for many locations. Filter to the specific locations you need.

Why this matters: LMP datasets like pjm_lmp_real_time_5_min contain 10,000+ pricing nodes. Querying all nodes for a single day can return millions of rows. If you only need hub prices for trading decisions, filter to those specific hubs.

Bad

# Returns data for all 10,000+ nodes
curl -H "Accept-Encoding: br, gzip" "https://api.gridstatus.io/v1/datasets/pjm_lmp_real_time_5_min/query?api_key=YOUR_API_KEY"

Good

# Returns data for one specific hub
curl -H "Accept-Encoding: br, gzip" "https://api.gridstatus.io/v1/datasets/pjm_lmp_real_time_5_min/query?\
start_time=2026-01-01&\
end_time=2026-01-02&\
filter_column=location&\
filter_value=WESTERN%20HUB&\
api_key=YOUR_API_KEY"

3

Select Only Needed Columns

Reduce response size by requesting only the columns you need.

Shell

# Only get timestamp, location, and LMP
curl -H "Accept-Encoding: br, gzip" "https://api.gridstatus.io/v1/datasets/caiso_lmp_real_time_5_min/query?\
start_time=2026-01-01&\
end_time=2026-01-02&\
columns=interval_start_utc,location,lmp&\
filter_column=location&\
filter_value=TH_SP15_GEN-APND&\
api_key=YOUR_API_KEY"

Python

# Only get timestamp, location, and LMP - skip energy, congestion, loss components
df = client.get_dataset(
    "caiso_lmp_real_time_5_min",
    start="2026-01-01",
    end="2026-01-02",
    columns=["interval_start_utc", "location", "lmp"],
    filter_column="location",
    filter_value="TH_SP15_GEN-APND"
)

# Response is ~60% smaller than without column selection

4

Request Compressed Responses

The API compresses responses with brotli or gzip whenever your client advertises support through the Accept-Encoding request header. For large dataset queries this shrinks the transfer by up to ~20x — a 60 MB JSON response drops to roughly 3 MB — which means faster downloads, lower bandwidth costs, and fewer timeouts.

Most HTTP clients request compression automatically, including the Grid Status Python client, requests, httpx, and web browsers. Set the header explicitly if your integration — for example, some ETL platforms or no-code HTTP connectors — does not send it by default.

Shell

# Send Accept-Encoding to receive a compressed response (brotli preferred, gzip fallback).
# curl --compressed sets this header and decompresses the response for you automatically.
curl -H "Accept-Encoding: br, gzip" "https://api.gridstatus.io/v1/datasets/ercot_lmp_by_settlement_point/query?\
start_time=2026-01-01&\
end_time=2026-01-02&\
filter_column=location&\
filter_value=HB_HOUSTON&\
limit=100&\
api_key=YOUR_API_KEY"

Python

# The Grid Status Python client requests compression automatically - no action needed.
# requests and httpx also send Accept-Encoding and decompress responses by default.
df = client.get_dataset(
    "ercot_lmp_by_settlement_point",
    start="2026-01-01",
    end="2026-01-02",
    filter_column="location",
    filter_value="HB_HOUSTON",
    limit=100,
)

5

Use Appropriate Limits

Set explicit limits to control data volume and costs.

Python

# Always set a limit during development
data = client.get_dataset(
    "ercot_fuel_mix",
    start="2026-01-01",
    end="2026-01-02",
    limit=1000  # Prevent accidental large queries
)

6

Resample Locally

If a query involving resampling is timing out on the API, download the raw data and perform resampling locally.

Pagination Strategies

Use Cursor Pagination for Large Datasets

Cursor-based pagination is more efficient than offset-based for large result sets. The client handles this automatically. For more see Pagination documentation.

Python

# The client handles pagination automatically - no manual cursor management needed
df = client.get_dataset(
    "ercot_fuel_mix",
    start="2026-01-01",
    end="2026-01-02"
)

# All pages are fetched and combined into a single DataFrame
print(f"Retrieved {len(df)} total rows")

Batch Large Date Ranges

For queries spanning months or years, batch into smaller chunks.

Python

import pandas as pd
from datetime import datetime, timedelta

def fetch_in_batches(
    client,
    dataset: str,
    start: datetime,
    end: datetime,
    batch_days: int = 7,
    **kwargs
) -> pd.DataFrame:
    """Fetch data in batches to avoid timeouts."""
    all_data = []
    current = start

    while current < end:
        batch_end = min(current + timedelta(days=batch_days), end)

        print(f"Fetching {current.date()} to {batch_end.date()}...")

        data = client.get_dataset(
            dataset,
            start=current.isoformat(),
            end=batch_end.isoformat(),
            **kwargs
        )

        if len(data) > 0:
            all_data.append(data)

        current = batch_end

    return pd.concat(all_data, ignore_index=True) if all_data else pd.DataFrame()

# Usage
df = fetch_in_batches(
    client,
    "ercot_fuel_mix",
    datetime(2026, 1, 1),
    datetime(2026, 2, 1),
    batch_days=7
)

Error Handling

Implement Retry Logic

Handle transient errors gracefully with exponential backoff.

Python

# The client handles retries automatically with exponential backoff

# Configure retry behavior when creating the client:
retry_client = gs.GridStatusClient(
    api_key="YOUR_API_KEY",
    max_retries=5,      # Retry up to 5 times
    base_delay=2.0,     # Start with 2 second delay
    exponential_base=2  # Double delay each retry
)

# All queries will automatically retry on transient errors (429, 500, 502, 503, 504)
df = retry_client.get_dataset("ercot_fuel_mix", start="2026-01-01", end="2026-01-02")

Monitor API Usage

Check usage before large operations to avoid hitting limits.

Python

def safe_query(client, dataset: str, expected_rows: int, **kwargs):
    """Check quota before querying."""
    usage = client.get_api_usage()

    rows_used = usage['current_period_usage']['total_api_rows_returned']
    rows_limit = usage['limits']['api_rows_returned_limit']
    rows_remaining = rows_limit - rows_used

    if rows_remaining < expected_rows:
        raise Exception(
            f"Insufficient quota: need ~{expected_rows:,}, "
            f"have {rows_remaining:,}"
        )

    return client.get_dataset(dataset, **kwargs)

# Usage
try:
    # Estimate: 288 5-min intervals/day * 1 location = ~288 rows
    df = safe_query(
        client,
        "ercot_fuel_mix",
        expected_rows=500,
        start="2026-01-01",
        end="2026-01-02"
    )
except Exception as e:
    print(f"Cannot proceed: {e}")

Data Quality

Data quality checks are critical for energy trading and operational applications where decisions are time-sensitive.

Verify Data Freshness

Check that data is current before using it in production.

Use case: Before executing trades based on current prices, verify the data is within an acceptable age threshold. Stale data could lead to trading on outdated price signals.

Python

from datetime import datetime, timezone, timedelta

def check_data_freshness(dataset_id: str, max_age_minutes: int = 30):
    """Check if dataset data is fresh enough."""
    metadata = client.get(
            f"{client.host}/datasets/{dataset_id}",
            return_raw_response_json=True
         )

    latest = datetime.fromisoformat(
        metadata['latest_available_time_utc'].replace('Z', '+00:00')
    )
    now = datetime.now(timezone.utc)
    age = now - latest

    if age > timedelta(minutes=max_age_minutes):
        print(f"Warning: Data is {age.total_seconds()/60:.0f} minutes old")
        return False

    return True

# Usage
if check_data_freshness("ercot_fuel_mix"):
    print("Data is fresh - proceeding with analysis")
else:
    print("Data may be stale - check for issues")

Validate Query Results

Verify that returned data meets expectations.

Python

import pandas as pd

def validate_results(df: pd.DataFrame, expected_cols: list, time_col: str = None):
    """Basic validation of query results."""
    errors = []

    # Check for expected columns
    missing_cols = set(expected_cols) - set(df.columns)
    if missing_cols:
        errors.append(f"Missing columns: {missing_cols}")

    # Check for empty result
    if len(df) == 0:
        errors.append("No data returned")

    # Check for time continuity (if time column specified)
    if time_col and time_col in df.columns:
        df[time_col] = pd.to_datetime(df[time_col])
        gaps = df[time_col].diff().dropna()

        # Check for unexpected gaps (>2x median interval)
        median_gap = gaps.median()
        large_gaps = gaps[gaps > median_gap * 2]

        if len(large_gaps) > 0:
            errors.append(f"Found {len(large_gaps)} time gaps")

    if errors:
        print("Validation errors:", errors)
        return False

    return True

# Usage
df = client.get_dataset("ercot_fuel_mix", start="2026-01-01", end="2026-01-02")

if validate_results(df, ['interval_start_utc', 'solar', 'wind'], 'interval_start_utc'):
    print("Data validated successfully")

Performance Summary

Practice	Impact	Implementation
Use time filters	High	Always set start_time and end_time
Filter by location	High	Use filter_column/filter_value
Resampling	High	Remove resampling. Fetch raw data first and resample locally.
Batch large requests	High	Split into smaller chunks
Retry on errors	High	Implement exponential backoff (included in the client)
Compress responses	High	Send Accept-Encoding: br, gzip (automatic in most clients)
Select columns	Medium	Use columns parameter
Cursor pagination	Medium	Use cursor instead of page
Monitor usage	Medium	Check quota before large queries

Related Documentation

• Advanced Query Features - Filtering, resampling, timezone

• Error Handling - Handle errors gracefully

• Utility Endpoints - API usage, metadata, column values