ClariGrid

# Clarigrid

Unified Python SDK for European and U.S. energy market data.

[![PyPI](https://img.shields.io/pypi/v/clarigrid)](https://pypi.org/project/clarigrid/)
[![Python](https://img.shields.io/pypi/pyversions/clarigrid)](https://pypi.org/project/clarigrid/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)

---

## What it is

Clarigrid provides a single, stable Python interface to access and normalise
European and U.S. energy market data from multiple sources. All data comes back as
timezone-aware pandas DataFrames with consistent column names and units.

Built-in free providers (no API key required) include **Energy-Charts**
(Europe), **Energinet** (DK1/DK2), **SMARD** (DE), **Elia** (BE), **NESO**
(GB), **Elexon/BMRS** (GB), and **ENTSOG** (EU gas).
Fingrid (FI), GIE AGSI/ALSI (European gas), and TenneT (NL) are also built in
and use free API keys.
For the United States, CAISO OASIS and NYISO provide no-auth market and system
data. EIA-930 provides nationwide hourly balancing-authority load, forecasts,
fuel generation, and physical interchange with a free EIA key.
Global historical meteorology and solar data are available from NASA POWER
without an API key.
ENTSO-E and other key-protected sources can be configured as described in
[API key setup](#api-key-setup) below.

---

## Install

```bash
pip install clarigrid
```

For the interactive setup wizard and CLI tools:

```bash
pip install clarigrid[auth]
```

---

## Quick start

```python
import clarigrid as cg

# Free providers — no key required.
cg.connect("smard")   # DE prices, load, generation
cg.connect("elia")    # BE load, generation
cg.connect("neso")    # GB load, embedded generation
cg.connect("elexon")  # GB prices, generation mix
cg.connect("entsog")  # EU gas flows (any TSO zone)
cg.connect("energycharts")  # European prices, power, forecasts and flows
cg.connect("energinet")  # DK1/DK2 prices, power, forecasts, flows and CO2
cg.connect("redata")  # ES load, generation, capacity and cross-border flows
cg.connect("rte")  # FR load, generation, forecasts, exchanges and CO2
cg.connect("fingrid")  # FI power, forecasts, flows, balancing and CO2 (free key)
cg.connect("gie")  # European gas storage and LNG inventory (free key)
cg.connect("eia")  # US balancing-authority load, generation and flows (free key)
cg.connect("caiso")  # CAISO day-ahead hub prices (no key)
cg.connect("nyiso")  # NYISO prices, load, forecasts and fuel mix (no key)
cg.connect("nasapower")  # Global daily/hourly weather and solar data (no key)

# Optional: set output timezone (default is UTC).
cg.set_timezone("Europe/Brussels")

# Fetch data — provider is chosen automatically by zone.
prices = cg.get_prices("DE", "2025-01-01", "2025-01-07")  # → smard
load   = cg.get_load("BE",   "2025-01-01", "2025-01-07")  # → elia
gen    = cg.get_generation("GB", "2025-01-01", "2025-01-07")  # → elexon
gas    = cg.get_gas_flows("BE-TSO-0001", "2025-01-01", "2025-01-07")  # → entsog
us_load = cg.get_load("CAISO", "2025-01-01", "2025-01-07")  # → eia (CISO)
np15 = cg.get_prices("CISO_NP15", "2025-01-01", "2025-01-07")  # → caiso
nyc = cg.get_prices("NYISO_NYC", "2025-01-01", "2025-01-07")  # → nyiso
weather = cg.get_weather(
    "40.7128,-74.0060",
    "2025-01-01",
    "2025-01-07",
    source="nasapower",
)
```

---

## API key setup

Some providers (ENTSO-E, TenneT) require a personal API key issued by
the upstream data source.  Clarigrid supports two ways to supply these keys.

### Option 1 — ClarigGrid account (recommended)

Store all your provider keys in one place at
[clarigrid.energy/saved](https://clarigrid.energy/saved).  The SDK then
fetches them automatically using a single **ClarigGrid API key**.

**First-time setup (interactive):**

```python
import clarigrid as cg
cg.connect("entsoe")
# Opens browser → log in at clarigrid.energy → keys fetched automatically.
```

Or use the CLI:

```bash
clarigrid setup          # guided wizard for all providers
clarigrid connect entsoe # authenticate a single provider
```

**Headless / CI environments:**  set one environment variable and no
browser is ever needed:

```bash
export CLARIGRID_API_KEY=your-clarigrid-uuid
```

The SDK uses `CLARIGRID_API_KEY` to fetch all your stored provider keys
from clarigrid.energy on the first `connect()` call of each session.

**How to get a `CLARIGRID_API_KEY`:**

1. Log in at [clarigrid.energy](https://clarigrid.energy)
2. Add your provider API keys at [clarigrid.energy/saved](https://clarigrid.energy/saved)
3. Run `cg.connect("entsoe")` once in an interactive terminal — the browser
   flow logs you in and stores your `CLARIGRID_API_KEY` locally.

---

### Option 2 — Manual key entry (no account needed)

If you prefer not to use a clarigrid.energy account, set provider keys
directly. Keys are stored in `~/.config/clarigrid/.env` (permissions: 600).

**Environment variable** (recommended for CI):

```bash
export ENTSOE_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
export TENNET_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
```

**Config file** — add to `~/.config/clarigrid/.env`:

```
ENTSOE_API_KEY="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
TENNET_API_KEY="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
```

## Zone routing

Each call to `cg.connect()` registers a provider and its capability-specific
zone coverage in
an internal router.  When you call `get_prices("DE")`, the router picks the
best connected provider for that zone and dataset automatically.

Multiple `connect()` calls accumulate coverage.  If two providers both cover
the same zone/dataset pair, the **later** `connect()` call wins.

```python
cg.connect("neso")    # covers GB: load, generation
cg.connect("elexon")  # covers GB: prices, generation — overwrites generation slot

# Now: GB prices → elexon, GB load → neso, GB generation → elexon
prices = cg.get_prices("GB", "2025-01-01", "2025-01-02")
load   = cg.get_load("GB",   "2025-01-01", "2025-01-02")
```

If no connected provider covers the requested zone/dataset, a helpful error
is raised:

```
ZoneNotCoveredError: No connected provider has 'prices' data for zone 'BE'.
  Consider: cg.connect('entsoe')
```

To bypass routing and force a specific provider:

```python
df = cg.get_load("GB", "2025-01-01", "2025-01-07", source="neso")
```

---

## Output format

All functions return a `pandas.DataFrame` with:

| Property | Value |
|---|---|
| Index | `DatetimeIndex` named `utc_time`, tz-aware |
| Timezone | UTC by default; change with `cg.set_timezone()` |
| Price column | `price_mwh` |
| Load column | `load_mw` |
| Generation columns | fuel-type specific, e.g. `solar_mw`, `wind_onshore_mw`, `nuclear_mw` |
| Gas flow column | `flow_kwh_d` |
| Gas storage | inventory in `*_mwh`; daily rates in `*_mwh_d` |
| LNG inventory | `*_thousand_m3`; send-out in `*_mwh_d` |
| Cross-border columns | signed MW; imports positive, exports negative |
| Installed capacity | `*_capacity_mw`; storage energy uses `*_energy_mwh` |
| Frequency | `frequency_hz` |
| Renewable shares | `*_pct` |

Price currency is stored in `df.attrs["currency"]` (for example ``"EUR"``,
``"GBP"``, or ``"USD"``):

```python
df = cg.get_prices("DE", "2025-01-01", "2025-01-07")
print(df.attrs["currency"])  # 'EUR'
```

European zone codes follow the ENTSO-E bidding zone convention (`BE`, `DE_LU`,
`FR` ...). U.S. electricity uses EIA/NERC balancing-authority codes (`CISO`,
`ERCO`, `PJM`, `NYIS`) and explicit market hubs (`CISO_NP15`). Common aliases
(`DE` → `DE_LU`, `CAISO` → `CISO`, `ERCOT` → `ERCO`) resolve automatically.

---

## Timezone

```python
cg.set_timezone("Europe/Brussels")   # all subsequent calls return Brussels time
cg.set_timezone("UTC")               # revert to default

df = cg.get_load("BE", "2025-01-01", "2025-01-07")
# df.index is tz-aware in Europe/Brussels
```

Data is always fetched and cached as UTC.  Timezone conversion is applied
at the output boundary only.

---

## Caching

Responses are cached locally at `~/.clarigrid/cache/` as Parquet files
(requires `pip install clarigrid[cache]`), keyed by provider + dataset +
zone + date range.  Historical data is cached indefinitely; live data
expires after 1 hour by default.

```python
from clarigrid.core import cache

cache.info()           # DataFrame showing cached entries
cache.clear()          # clear all
cache.clear("smard")   # clear one provider
cache.set_live_ttl(1800)  # change live-data TTL to 30 min
```

Disable caching per call:

```python
df = cg.get_prices("DE", "2025-01-01", "2025-01-07", use_cache=False)
```

---

## CLI reference

Requires `pip install clarigrid[auth]`.

```bash
clarigrid setup                    # guided wizard — configure all providers
clarigrid connect <source>         # authenticate a single provider
clarigrid auth --show              # list configured sources (keys masked)
clarigrid auth --clear <source>    # remove key for a specific source
clarigrid auth --clear --all       # remove all stored keys
```

---

## API reference

| Function | Description |
|---|---|
| `cg.connect(provider)` | Connect provider; handles auth for key-guarded sources |
| `cg.set_timezone(tz)` | Set output timezone (IANA string, default `"UTC"`) |
| `cg.get_prices(zone, start, end, market="day_ahead", node=None)` | Electricity prices → `price_mwh`; optional node for nodal markets |
| `cg.get_load(zone, start, end)` | Actual total load → `load_mw` |
| `cg.get_generation(zone, start, end)` | Generation per fuel type → `*_mw` columns |
| `cg.get_generation_forecast(zone, start, end)` | Wind/solar generation forecast → `*_forecast_mw` |
| `cg.get_load_forecast(zone, start, end)` | Load forecast → `load_forecast_mw` |
| `cg.get_physical_flows(zone, start, end)` | Signed cross-border physical flows in MW |
| `cg.get_commercial_schedule(zone, start, end)` | Signed commercial exchanges in MW |
| `cg.get_installed_capacity(zone, start, end)` | Installed power by technology in MW |
| `cg.get_frequency(zone, start, end)` | System frequency → `fr