datapulse-my

# DataPulse

**Live dashboard:** https://www.data-pulse.my

**Open in Google Colab:** [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/r3dz4r/datapulse-my/blob/main/docs/trust-layer-notebook.ipynb)

[![datapulse-my MCP server](https://glama.ai/mcp/servers/r3dz4r/datapulse-my/badges/score.svg)](https://glama.ai/mcp/servers/r3dz4r/datapulse-my)
[![M8ven Verified](https://m8ven.ai/badge/mcp/r3dz4r-datapulse-my-fsfgq3?variant=verified)](https://m8ven.ai/mcp/r3dz4r-datapulse-my-fsfgq3)
[![mcpgrade](https://img.shields.io/badge/mcpgrade-100%2F100%20(Grade%20A)-success?style=flat&logo=anthropic)](https://www.npmjs.com/package/mcpgrade)
<!-- m8ven-verify: d1505f0f7e0429963789e95995216ca3 -->

> **🤖 AI-agent-ready** — Wire DataPulse into Claude Desktop, Cursor, Cline, or
> any MCP-compatible client with one config block. Your agent gets
> <!-- BEGIN readme-hero -->
**418 official Malaysian datasets** — including **30 GTFS transit feeds (KTMB,
Prasarana, BAS.MY)** — with declared licences and an honest ten-status trust
taxonomy instead of a blanket green checkmark.
<!-- END readme-hero -->
>
> → [Connect your AI agent in 30 seconds](#connect-an-ai-agent)

## This is DataPulse

When an AI quote is wrong, it is often wrong because the **underlying data was
stale, mis-licensed, or unverifiable** — not because the model hallucinated.
An official-looking page does not tell an agent when the dataset behind it last
updated, who published it, whether it may legally be reused, or whether the
observation can be reproduced by a second party.

DataPulse exists to make that uncertainty explicit. It is an open, read-only
**verification layer for Malaysian public data**: it continuously probes
<!-- BEGIN readme-cover -->**418 official datasets**<!-- END readme-cover -->,
and publishes — for each one — machine-readable *evidence* about whether the
source is reachable, how fresh the content is, what licence applies, how the
schema behaves, and when the observation was signed.

It does **not** replace the official source. It documents, on an honest and
reproducible basis, what the official portal states and whether that material is
current, so you know what you are reusing or citing. The verification speaks
for itself: every claim here is a live, checkable artefact, not a promise.

## What we do, simply

- **We watch the sources.** A scheduled probe revisits each dataset under its
  declared cadence and records what it actually finds — reachability, an honest
  freshness signal, schema shape, record counts, and collection quirks.
- **We state the truth plainly.** Instead of a blanket green checkmark, each
  dataset carries one of ten honest health statuses (`fresh`, `aging`, `stale`,
  `discontinued`, `degraded`, `browser-dependent`, `unreachable`, `unknown`,
  `unknown-freshness`, `reference`). A dataset that cannot be proven fresh is
  labelled `unknown-freshness` — not silently treated as healthy.
- **We publish evidence, not just claims.** Each dated observation is signed
  and recorded to an immutable public log, so you can verify *when* DataPulse
  observed the source and that the record has not been altered.
- **We make it machine-readable first.** The whole portfolio is discoverable
  from one index and queryable over a read-only MCP server, so an agent receives
  the same freshness, licence, and provenance signal a careful human reviewer
  would.

## Who this serves

- **AI builders and agent developers**, who want a model to check a Malaysian
  figure's freshness and licence before it cites the number — without building a
  bespoke integration or trusting a scraping pipeline.
- **Researchers, analysts, and journalists**, who need to ground coursework,
  a thesis, a dashboard, or a published figure in data whose currency and licence
  they can actually verify.
- **Compliance and regulatory-monitoring teams**, who must keep a tamper-evident
  trail that an official figure was checked at a known time before it reached a
  product or a public statement.
- **Civic technologists and public servants**, who want a transparent,
  reproducible view of how discoverable and reliably described public data is.

## Why you can trust the verification

Three independent, checkable layers. You do not have to take DataPulse's word —
you can verify each with the published public key, the public Git source record,
and the public transparency log:

| Layer | What it proves | How to check it yourself |
|---|---|---|
| **Signed envelope** | Each per-dataset observation is **Ed25519-signed** over its exact content by a key in the published registry | `python3 scripts/verify_external.py` |
| **Source of record** | The served observation **byte-matches** the versioned Git source | `python3 scripts/verify_external.py` |
| **Temporal witness** | The health statement carries a **Rekor/Sigstore public-log inclusion proof** | `python3 scripts/verify_external.py` |

Run it yourself, from anywhere, with no checkout and no DataPulse code:

```bash
curl -fsSLO https://raw.githubusercontent.com/r3dz4r/datapulse-my/main/scripts/verify_external.py
python3 verify_external.py
```

See [Verify DataPulse externally](docs/verify-datapulse-externally.md) for the
full guide, and [our methodology](#dataset-health) below for how health is judged.

A verification layer is only as honest as its method, so DataPulse deliberately
tells you **when it cannot be sure** — a source that cannot be proven current is
labelled accordingly, never silently marked healthy. That is the boundary we
hold: the platform proves the integrity and timing of its *observations*, not
that an upstream government figure is semantically true. That distinction is the
whole point of an evidence layer, and we do not blur it.

## Dataset health

Health is reported as `fresh`, `aging`, `stale`, `discontinued`, `degraded`,
`browser-dependent`, `unreachable`, `unknown`, `unknown-freshness`, or
`reference`. Unknown freshness means the URL and content shape work, but neither
a Last-Modified header nor a parseable content date proves when the data was
updated. Reference means versioned lookup data is reachable and its record count
is measured, while date-based freshness does not apply. Within the catalogue,
`data_type` refines the reference family without changing the status: `policy-reference`
rows (policy state that stays valid until superseded — BNM OPR is current while
unchanged, not stale) and `reference-current` rows (lookups that must still pass
freshness, such as a bank-rate table that can itself go stale) are judged by their
declared policy, while plain `reference` rows are static. The public
[`_trust_summary`](health/latest.json) shows the distribution and explicitly
counts missing freshness and row-count signals.

**Discontinued** — The source has stopped publishing new data. The data is
frozen at the last known content date. This is not a freshness failure — it's a
publisher decision.

<!-- BEGIN readme-health -->
Current distribution (`_trust_summary`): [90 fresh](badges/status-fresh.svg) · [113 aging](badges/status-aging.svg) · [187 stale](badges/status-stale.svg) · [1 discontinued](badges/status-discontinued.svg) · [1 degraded](badges/status-degraded.svg) · [5 browser-dependent](badges/status-browser-dependent.svg) · [7 unknown-freshness](badges/status-unknown-freshness.svg) · [14 reference](badges/status-reference.svg)
<!-- END readme-health -->

**Subscribe:** [RSS feed](feed.xml) — get notified when dataset health changes.

### Browser-dependent datasets

Five sources (currently 1.4% of the catalogue) require a real browser to probe because their
source pages render client-side JavaScript: `eperolehan-diklankan`,
`doe_apims`, `doe_rqims`, `doe_mqims`, and `kkm_idengue`.

DataPulse uses **[Camofox](https://github.com/jo-inc/camofox-browser)**, a
self-hosted patched headless-Chromium sidecar, to probe these. The probe path
is [`check.sh`](scripts/check.sh) → Camofox sidecar → DOM snapshot →
content-date extraction.

**To enable browser probing:**

1. Run the Camofox Docker sidecar on a reachable address (default
   `http://localhost:9377`). The probe script and the GitHub Actions
   workflow pick this up from the `CAMOFOX_BASE_URL` environment
   variable; nothing in this repo encodes a public IP.
2. Set `CAMOFOX_BASE_URL` to that address.
3. Restart the timer with `systemctl restart datapulse-health.timer`.

Without Camofox, those five datasets will sit at `browser-dependent` — the
**honest** status: DataPulse cannot probe them without a browser, so it says
so rather than failing silently. See
[`scripts/smoke_browser_probes.sh`](scripts/smoke_browser_probes.sh) for
isolated smoke tests.

## Methodology

| Topic | DataPulse's position |
|---|---|
| **Health status** | Ten-status taxonomy, judged by reachability + an honest freshness signal (`Last-Modified`, parseable content date, or declared policy) — never a fabricated green checkmark. A series that stopped publishing is `discontinued` (a publisher decision, frozen data), not a freshness failure. |
| **Licence** | Every dataset declares its licence machine-readably. <!-- BEGIN readme-licences -->Creative Commons Attribution 4.0 (285); MBPP Government Open Data Terms (attribution required) (1); MIT License (8); Open Government Licence (Malaysia) (115); Publisher licence not stated; portal disclaimer applies (4); Singapore Open Data Licence v1.0 (attribution required) (5).<!-- END readme-licences --> A second party can reproduce this from `datapulse.json` → `.datasets[].licence`. |
| **Freshness cadence** | Each dataset is probed on its own tiered schedule (5-minute timer, cadence-aware) — `daily` references, `weekly` fuel prices, `monthly` surveys, etc. Always with the human-readable `steward` and a stable `custodian` ID for publisher provenance. |
| **Provenance** | Stable `custodian` per dataset; signed probe attestations per observation |
| **Observed claim** | The platform