morocco-communes

# Morocco communes

An open dataset and HTTP API for Morocco's administrative divisions: 12 régions, 75
provinces and préfectures, 8 préfectures d'arrondissements, 213 cercles, 1,503 communes
and 41 arrondissements, with
official HCP geographic codes, names in French and Arabic, 2024 and 2014 population, HCP's
census figures on age, education, languages, work and housing from both years, the 2024
count of economic establishments, and boundaries from OpenStreetMap.

HCP publishes the census as spreadsheets. This builds a dataset, an API and a site from them
and from OpenStreetMap, and shows the working.

## The dataset

`data/v1/` is committed and versioned, so you can use it without the API at all.
[`CHANGELOG.md`](CHANGELOG.md) says what each version added; the build refuses a version
with no entry in it.

| Directory | Holds | Licence |
|---|---|---|
| `attributes/` | every unit, JSON and CSV | HCP, on CC BY 4.0 terms |
| `geometry/` | one TopoJSON per région, and which communes border which | **ODbL**, share-alike |
| `indicators/` | the census indicators for every unit, 2024 and 2014, JSON and CSV | HCP, on CC BY 4.0 terms |
| `economy/` | the 2024 count of economic establishments for every unit, JSON and CSV | HCP, on CC BY 4.0 terms |
| `housing/` | the 2024 urban housing stock, by dwelling, JSON and CSV | HCP, on CC BY 4.0 terms |
| `crosswalk/` | the 2014 ↔ 2024 reconciliation | HCP, on CC BY 4.0 terms |
| `insights/` | each commune's 2024 figures that stand out, with its neighbours and 2014 beside each, JSON | the lines this repository's own; the figures HCP, on CC BY 4.0 terms |
| `sources.json` | each source's digest, licence and vintage | |

The licences differ by directory and `geometry/` carries its own LICENSE. Five fields on
each commune (`centroid`, `bbox`, `osm`, `areaKm2` and `density`) come from OpenStreetMap
and are ODbL too; `provenance.geometry` marks them. Every level has a CSV that opens
cleanly in Excel, Arabic included.

Rebuild it with `pnpm dataset:build`. From the same cache the output is byte-identical.

`pnpm build` also writes the boundaries as GeoJSON from the TopoJSON above: each commune,
and each province and région outlined from its communes. They're served beside it rather
than committed.

## The API

Three tiers, and which one served a response is in its `X-Api-Tier` header.

**Pre-rendered.** 11,629 files written at build time and served straight from
Cloudflare Pages, without running any code. Free and unmetered.

```
GET /api/regions.json                        GET /api/regions/01.json
GET /api/regions/01/provinces.json           GET /api/regions/01/communes/page/1.json
GET /api/provinces/01.511.json               GET /api/provinces/01.511/cercles.json
GET /api/cercles/01.511.05.json              GET /api/communes/01.511.01.0.json
GET /api/communes/page/1.json                GET /api/communes/type/urban/page/1.json
GET /api/communes/01.511.01.0/arrondissements.json
GET /api/communes/09.581.01.07/neighbours.json
GET /api/communes/01.511.01.0/indicators.json
GET /api/communes/09.581.01.07/economy.json
GET /api/arrondissements/01.511.01.05.json   GET /api/indicators.json
GET /api/economy.json                        GET /api/version.json
GET /data/v1/**
```

**Alias.** The query-string and extensionless shapes. A path-keyed asset store can't
match on a query string, so the Worker resolves these to the file that already holds the
answer and names it in `Content-Location`:

```
GET /api/communes?province=01.511&page=1     GET /api/communes?type=urban
GET /api/communes/tanger                     GET /api/communes/001511010
GET /api/communes/tanger/indicators          GET /api/communes/tiznit/economy
```

**Computed.** The answers no file holds:

```
GET /api/search?q=tanger&levels=commune&limit=10
GET /api/communes/near?lat=33.5731&lng=-7.5898&radius=15
GET /api/communes/at?lat=35.786&lng=-5.8125
GET /api/communes?province=01.511&type=urban
GET /api/communes?sort=-population&min_population=100000
GET /api/communes?region=01&sort=-labour.unemploymentRate
GET /api/communes?sort=-economy.establishments.jobs
```

A list sorts by any of the census indicators, by its path, and each commune it lists then
carries the figure it was sorted by. `2014.` before the path sorts by the 2014 figure and
`change.` by how far a commune moved between the censuses. An establishment count goes
under `economy.`.

Search takes French, Arabic, a slug or a code. It folds the alef variants, ta-marbuta and alef
maqsura the names actually carry, and the tatweel and vowel marks they never do but people
type anyway. It matches names spelt another way by their consonants, so `titwan` finds
Tétouan and `jdida` El Jadida. `01.511.01.0`, `001511010`, `1511010` and `tanger` all
address one commune, and 18 exonyms are listed by hand because no amount of character folding gets from `Fez`
to Fès or from `Port Lyautey` to Kénitra.

**What's counted.** The alias and computed routes and the MCP server count their own use
in Workers Analytics Engine: the route with its codes taken out, or an MCP message's tool
when it called one and its method otherwise, the name an MCP client gives itself, the
first word of the User-Agent, the country, the status and how long it took. Pre-rendered
files run no code, so they aren't counted. `pnpm stats` reads it back, with the npm, PyPI
and GitHub numbers.

A second store, in D1, counts 5 kinds of row: a search, a place opened, a tool an assistant
calls, the name an MCP client gives itself, and a file downloaded. An assistant connecting
through the MCP server is counted by the name it gives itself, and each tool it calls by
the tool's name and the place it asks about. A row is a day and a handful of categories,
and no row holds a key, a cookie or an address that ties it to another. `pnpm demand`
reads it back; the site's `/docs/privacy/` page says exactly what's kept.

`pnpm deploy:live` switches it on. It applies the migration to the live database, builds
the site with the most looked-up page read from that database, deploys it, then deploys
the nightly rollup Worker. A plain `pnpm build` doesn't query the database and leaves that
page empty. A failed write is swallowed, so `pnpm demand` a day after the first deploy is
how to see rows are arriving.

Every response is enveloped with `data`, `meta` and `links`. Errors are RFC 9457 problem
documents, and a well-formed code that names nothing is a 404 while something unsearchable
is a 400. CORS is open.

Full reference: [`api/README.md`](api/README.md).

### For programs and agents

- **`/api/openapi.json`**: an OpenAPI 3.1 description of every route, built from the
  same module the Worker reads its limits from, so the defaults and bounds it states are
  the ones enforced. Most agent frameworks turn it into tools directly.
- **`/llms.txt`**: a short markdown map of the API and the dataset, in the llmstxt.org
  shape, for an LLM reading the site.
- **`/mcp`**: an MCP server with 10 read-only tools (`search`, `get_commune`, `get_unit`,
  `communes_near`, `commune_at`, `list_communes`, `get_indicators`, `get_economy`,
  `get_housing`, `get_insights`), so
  Claude, Claude Code and other MCP clients can query the data directly, census figures
  included. The site's `/docs/mcp/` page has the setup for each client.

### In a form, or offline

- **`/components/commune-picker.js`**: a custom element that fills a région, a province
  and a commune `<select>` from the static files, so a form posts the commune's HCP code.
  It's served with open CORS, and `/docs/components/` has a React version beside it.
- **`morocco-communes`**: the dataset as an npm package, in `packages/morocco-communes/`.
  Codes, names, parents and population as ES modules with types, one per level, and
  without the OpenStreetMap fields, so it carries no ODbL terms. `pnpm npm:build` builds
  it from `data/v1`, and the package takes the dataset's version number.
- **`morocco-communes` on PyPI**: the same dataset for analysts, in
  `packages/morocco-communes-py/`. The units, both censuses, the establishments and the
  crosswalk, each table a pandas DataFrame, or a list of dicts with `as_frame=False`,
  which needs nothing but the standard library. `pnpm py:build` builds its data from
  `data/v1`, gzipped, and it takes the dataset's version too.

## The docs site

`site/` is an Astro site in English and French, with a light and dark theme and a control
to pick either or follow the system. It builds to static HTML: the home page, 7 docs pages
(the API reference, the MCP setup, the components, the npm package, the Python package,
the census figures and a glossary), and a page for every région, province and commune,
3,216 pages in all.

The home page opens on a map of every commune, shaded by density, change since 2014, or
urban and rural. Hovering one shows its figures and clicking opens its page. A commune's
page has its figures and rank, a map of its province, the communes it borders, where its
change sits among all of them, and its census figures: an age pyramid of men and women,
headline rates beside the women's and the country's, and the languages its people use.
A list of every commune filters as you type, in French or Arabic.

Its JavaScript is the map's hover and switch, the list's filter, the playground (a Solid
island that queries whatever API it's deployed beside and shows the `X-Api-Tier` of each
response), the picker on the components page, and the copy buttons.

Everything on it that describes the data or the API is generated at build time, by the
scripts under `site/scripts/`:

- `map.ts` draws the home map from the TopoJSON the API serves. Each shared border is one
  arc, simplified once for both of its communes, so the fills meet without slivers. Its
  colours were checked for contrast and colour-blind separation in both themes.
- `hierarchy.ts` builds the code ladder, a real chain from région down to arrondissement.
- `charts.ts` computes population change per r