datapulse-my

# DataPulse MY

**Live dashboard:** https://r3dz4r.github.io/datapulse-my/

> **🤖 AI-agent-ready** — DataPulse MY is built for AI. An agent fetches one file
> ([`llms.txt`](https://r3dz4r.github.io/datapulse-my/llms.txt)) and can use all
> **122 official Malaysian datasets** immediately — including **+ 30 GTFS transit
> feeds (KTMB, Prasarana, BAS.MY)** — no integration, no scraping,
> licences declared, with an honest eight-status trust taxonomy instead of a
> blanket green checkmark.

DataPulse MY is an open-source trust layer for Malaysian public data. It makes
official datasets easier to assess and reuse by publishing a small manifest,
human-readable health reports, and machine-readable health envelopes.

It does not replace the official source. It documents what is available,
whether it is fresh, how its schema behaves, and which collection quirks users
need to handle.

## Who it is for

- Journalists and researchers checking whether a public dataset is usable.
- Civic technologists building reproducible data pipelines.
- Public servants improving the discoverability and reliability of open data.
- Developers who need stable, machine-readable dataset health metadata.

## Use this for

- **Journalist fact-check:** before citing a fuel price figure, check fuelprice
  freshness to make sure it's current.
- **Pipeline health gate:** fail the build if data.gov.my has been down for
  more than 24 hours.
- **RAG knowledge base:** consume the JSON envelopes as structured context for
  a chatbot answering "what's the latest BNM rate?"

## Dataset health

Health is reported as `fresh`, `aging`, `stale`, `degraded`,
`browser-dependent`, `unreachable`, `unknown`, or `unknown-freshness`. The last
value means the URL and content shape work, but neither a Last-Modified header
nor a parseable content date proves when the data was updated. The public
[`_trust_summary`](health/latest.json) shows the distribution and explicitly
counts missing freshness and row-count signals.

Current distribution (`_trust_summary`): [80 fresh](badges/status-fresh.svg) · [25 aging](badges/status-aging.svg) · [11 stale](badges/status-stale.svg) · [5 browser-dependent](badges/status-browser-dependent.svg) · [1 unreachable](badges/status-unreachable.svg)

<!--
Statuses with zero count are omitted. Full per-dataset health is in
[health/latest.json](health/latest.json) and per-dataset badges live in
[badges/](badges/).
-->

**Subscribe:** [RSS feed](feed.xml) — get notified when dataset health changes.

## AI-agent-ready — what it means for you

Give your organisation's AI tools current, licensed, and verified Malaysian
public data without first building a custom integration. DataPulse MY makes the
full portfolio discoverable from one self-describing index, ready for agents,
RAG systems, and internal knowledge tools to consume.

**What being AI-ready gives you**

- **Zero integration work:** an AI agent or LLM/RAG system fetches one
  [`llms.txt`](https://r3dz4r.github.io/datapulse-my/llms.txt) and can use the
  entire portfolio immediately — no scraping, API-key setup, or data-format
  reverse-engineering.
- **Honest freshness signals:** automatic weekly monitoring separates HTTP
  reachability, browser dependency, schema validity, and source freshness so
  missing evidence is visible instead of being labelled healthy.
- **Machine-readable and licence-clear:** every dataset has a JSON envelope with
  its schema, licence, and refresh cadence, giving legal and engineering teams
  the information they need to approve and integrate it.
- **Trustworthy for AI:** verified official sources and explicit licences let
  agents cite and use the data without permission ambiguity.
- **RAG and knowledge-base ready:** drop the envelopes into a retrieval pipeline
  to ground chatbots and AI tools in current Malaysian public data.

**122 datasets, 110 under CC BY 4.0, 12 under OGL, each assessed with the honest
eight-status trust taxonomy.**

### MCP server (read-only)

DataPulse MY also exposes an AI-ready, read-only MCP server so agents can query
the catalogue natively:

- Endpoint: `https://mcp.data-pulse.my/mcp` (Streamable HTTP, no auth)
- 5 tools: `search_datasets`, `get_dataset`, `find_stale`, `get_provenance`, `find_by_licence`
- 3 resources: `datapulse://index`, `datapulse://licences`, `datapulse://{dataset_id}`

The public endpoint is live and serves all 5 read-only tools over the
122-dataset catalogue.

Connect from Claude Desktop:

```json
{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}
```

See [`llms.txt`](https://r3dz4r.github.io/datapulse-my/llms.txt) for the full
discovery index, and [`docs/mcp-deploy.md`](./docs/mcp-deploy.md) for the
deployment architecture.

### How to consume the data

Verify access:

```sh
curl -s https://r3dz4r.github.io/datapulse-my/llms.txt
```

- [`llms.txt`](https://r3dz4r.github.io/datapulse-my/llms.txt) — curated dataset index
- [`datapulse.json`](https://r3dz4r.github.io/datapulse-my/datapulse.json) — manifest with a declared `$schema`
- [`health/latest.json`](https://r3dz4r.github.io/datapulse-my/health/latest.json) — latest freshness snapshot
- [`feed.xml`](https://r3dz4r.github.io/datapulse-my/feed.xml) — dataset health change feed
- [`datapulse.schema.json`](https://r3dz4r.github.io/datapulse-my/datapulse.schema.json) — manifest schema

To consume the portfolio:

1. Fetch [`llms.txt`](https://r3dz4r.github.io/datapulse-my/llms.txt) for the curated index.
2. Fetch [`datapulse.json`](https://r3dz4r.github.io/datapulse-my/datapulse.json)
   for the machine-readable manifest, including licence, refresh cadence, and
   geographic coverage.
3. Fetch [`health/latest.json`](https://r3dz4r.github.io/datapulse-my/health/latest.json)
   to check freshness before use.
4. Cite each dataset according to its licence and attribution requirements.

[`robots.txt`](https://r3dz4r.github.io/datapulse-my/robots.txt) allows all agents;
[`scripts/verify_agent_ready.sh`](https://github.com/r3dz4r/datapulse-my/blob/main/scripts/verify_agent_ready.sh)
is the agent-consumer self-test.

## Included datasets

- [Malaysian Fuel Prices](data/fuelprice.md) — Samples:
  [CSV](samples/fuelprice.csv), [JSON](samples/fuelprice.json)
- [ePerolehan Tender Notices (DIIKLANKAN)](data/eperolehan-diklankan.md) —
  [Sample JSON](samples/eperolehan-diklankan.json)
- [PriceCatcher (Daily Grocery Prices)](data/pricecatcher.md) — Samples:
  [main CSV](samples/pricecatcher.csv),
  [item lookup](samples/pricecatcher_lookup_item.csv),
  [premise lookup](samples/pricecatcher_lookup_premise.csv)

### Daily Reference Data (Bank Negara Malaysia)

These four BNM reference-rate datasets are updated on weekdays at fixed MYT
publication times:

- [BNM Daily Exchange Rates (0900)](data/exchangerates_daily_0900.md) —
  [Sample JSON](samples/exchangerates_daily_0900.json)
- [BNM Daily Exchange Rates (1130)](data/exchangerates_daily_1130.md) —
  [Sample JSON](samples/exchangerates_daily_1130.json)
- [BNM Daily Exchange Rates (1200)](data/exchangerates_daily_1200.md) —
  [Sample JSON](samples/exchangerates_daily_1200.json)
- [BNM Daily Exchange Rates (1700)](data/exchangerates_daily_1700.md) —
  [Sample JSON](samples/exchangerates_daily_1700.json)

- [MET Malaysia Weather Forecast](data/met_weather.md) — Samples:
  [CSV](samples/met_weather.csv), [JSON](samples/met_weather.json)
- [DOE APIMS Air Quality (Hourly)](data/doe_apims.md) —
  [Sample JSON](samples/doe_apims.json)
- [DOE RQIMS River Water Quality (Continuous)](data/doe_rqims.md) —
  [Sample JSON](samples/doe_rqims.json)
- [DOE MQIMS Marine Water Quality (Monthly)](data/doe_mqims.md) —
  [Sample JSON](samples/doe_mqims.json)
- [KKM iDengue Weekly Dengue Cases](data/kkm_idengue.md) —
  [Sample JSON](samples/kkm_idengue.json)
- [OpenDOSM Crime by District (Annual)](data/dosm_crime_district.md) —
  [Sample JSON](samples/dosm_crime_district.json)
- [OpenDOSM Monthly CPI by State](data/dosm_cpi_state.md) —
  [Sample JSON](samples/dosm_cpi_state.json)
- [OpenDOSM Annual Real GDP by State](data/dosm_gdp_state_real_supply.md) —
  [Sample JSON](samples/dosm_gdp_state_real_supply.json)
- [OpenDOSM Quarterly Real GDP](data/dosm_gdp_qtr_real.md) —
  [Sample JSON](samples/dosm_gdp_qtr_real.json)
- [OpenDOSM Annual Real GDP by Supply Sector](data/dosm_gdp_annual_real_supply.md) —
  [Sample JSON](samples/dosm_gdp_annual_real_supply.json)
- [OpenDOSM Monthly Trade Headline](data/dosm_trade_headline.md) —
  [Sample JSON](samples/dosm_trade_headline.json)
- [OpenDOSM Monthly CPI Inflation by Division](data/dosm_cpi_inflation.md) —
  [Sample JSON](samples/dosm_cpi_inflation.json)
- [OpenDOSM Monthly Trade by End Use (BEC)](data/dosm_trade_enduse_bec.md) —
  [Sample JSON](samples/dosm_trade_enduse_bec.json)
- [OpenDOSM Quarterly Labour Force Statistics](data/dosm_lfs_qtr.md) —
  [Sample JSON](samples/dosm_lfs_qtr.json)
- [OpenDOSM Quarterly Labour Force Statistics by State](data/dosm_lfs_qtr_state.md) —
  [Sample JSON](samples/dosm_lfs_qtr_state.json)
- [OpenDOSM Annual Employment by Sector and Sex](data/dosm_employment_sector.md) —
  [Sample JSON](samples/dosm_employment_sector.json)
- [OpenDOSM Annual Population by State](data/dosm_population_state.md) —
  [Sample JSON](samples/dosm_population_state.json)
- [OpenDOSM Annual Nominal GDP by Supply Sector](data/dosm_gdp_annual_nominal_supply.md) —
  [Sample JSON](samples/dosm_gdp_annual_nominal_supply.json)
- [OpenDOSM Quarterly Nominal GDP](data/dosm_gdp_qtr_nominal.md) —
  [Sample JSON](samples/dosm_gdp_qtr_nominal.json)
- [OpenDOSM Quarterly Real GDP (Seasonally Adjusted)](data/dosm_gdp_qtr_real_sa.md) —
  [Sample JSON](samples/dosm_gdp_qtr_real_sa.json)
- [OpenDOSM Annual Nominal GDP and GNI](data/dosm_gdp_gni_annual_nominal.md) —
  [Sample JSON](samples/dosm_gdp_gni_annual_nominal.json)
- [OpenDOSM Monthly Core CPI Inflation by Division](data/dosm_cpi_core_inflation.md) —
  [Sample JSON](samp