datapulse-my

DataPulse

Live dashboard: https://www.data-pulse.my

Open in Google Colab:

🤖 AI-agent-ready — Wire DataPulse into Claude Desktop, Cursor, Cline, or any MCP-compatible client with one config block. Your agent gets

440 official Malaysian datasets — including 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY) — with declared licences and an honest ten-status trust taxonomy instead of a blanket green checkmark.

→ Connect your AI agent in 30 seconds

This is DataPulse

When an AI quote is wrong, it is often wrong because the underlying data was stale, mis-licensed, or unverifiable — not because the model hallucinated. An official-looking page does not tell an agent when the dataset behind it last updated, who published it, whether it may legally be reused, or whether the observation can be reproduced by a second party.

DataPulse is an open, read-only verification layer for Malaysian public data. It continuously observes Malaysia's official open datasets — those published through data.gov.my, Bank Negara Malaysia, DOSM, the Department of Environment, the Ministry of Health, KPDN and MET Malaysia — and publishes machine-readable evidence about each one: whether the source is reachable, how fresh its content is, which licence applies, whether its structure or record count has changed, and when the observation was signed. Every dataset carries one of ten explicit statuses instead of a blanket green tick, and each published observation can be checked by a third party: receipts are signed, the signing keys are published with validity windows and rotation history, and a single-file verifier reproduces the check with no DataPulse installation. DataPulse does not replace the official source and does not certify that a publisher's data is substantively correct — it makes the condition of the source observable, and the observation reproducible.

What we do, simply

• We watch the sources. A scheduled probe revisits each dataset under its declared cadence and records what it actually finds — reachability, an honest freshness signal, schema shape, record counts, and collection quirks.

• We state the truth plainly. Instead of a blanket green checkmark, each dataset carries one of ten honest health statuses (fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, reference). A dataset that cannot be proven fresh is labelled unknown-freshness — not silently treated as healthy.

• We publish evidence, not just claims. Each dated observation is signed and recorded to an immutable public log, so you can verify when DataPulse observed the source and that the record has not been altered.

• We make it machine-readable first. The whole portfolio is discoverable from one index and queryable over a read-only MCP server, so an agent receives the same freshness, licence, and provenance signal a careful human reviewer would.

Who this serves

• AI builders and agent developers, who want a model to check a Malaysian figure's freshness and licence before it cites the number — without building a bespoke integration or trusting a scraping pipeline.

• Researchers, analysts, and journalists, who need to ground coursework, a thesis, a dashboard, or a published figure in data whose currency and licence they can actually verify.

• Compliance and regulatory-monitoring teams, who must keep a tamper-evident trail that an official figure was checked at a known time before it reached a product or a public statement.

• Civic technologists and public servants, who want a transparent, reproducible view of how discoverable and reliably described public data is.

Why you can trust the verification

Three independent, checkable layers. You do not have to take DataPulse's word — you can verify each with the published public key, the public Git source record, and the public transparency log:

Run it yourself, from anywhere, with no checkout and no DataPulse code:

curl -fsSLO https://raw.githubusercontent.com/r3dz4r/datapulse-my/main/scripts/verify_external.py
python3 verify_external.py

See Verify DataPulse externally for the full guide, and our methodology below for how health is judged.

Verify an observation receipt

scripts/observation_verify.py verifies receipt identity, signature, key registry validity, and any available chain linkage from local files. It makes no network calls in any mode. Supplying --health additionally reproduces the normalized artifact digest and re-derives its dataset and freshness-status claims: only health_binding: checked and artifact_claims: verified attest those claims. Without --health, the report explicitly says artifact_claims: NOT verified; use --require-health-binding when that incomplete result must fail.

For receipts with a signed @ artifact locator, pass --repo owner/name to print the corresponding immutable raw-GitHub URL. The repository is supplied, never guessed or hardcoded. Fetch that URL separately, then give the saved file to --health; the URL is a discovery aid, not proof until the local digest and claims check succeeds.

A verification layer is only as honest as its method, so DataPulse deliberately tells you when it cannot be sure — a source that cannot be proven current is labelled accordingly, never silently marked healthy. That is the boundary we hold: the platform proves the integrity and timing of its observations, not that an upstream government figure is semantically true. That distinction is the whole point of an evidence layer, and we do not blur it.

Dataset health

Health is reported as fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, or reference. Unknown freshness means the URL and content shape work, but neither a Last-Modified header nor a parseable content date proves when the data was updated. Reference means versioned lookup data is reachable and its record count is measured, while date-based freshness does not apply. Within the catalogue, data_type refines the reference family without changing the status: policy-reference rows (policy state that stays valid until superseded — BNM OPR is current while unchanged, not stale) and reference-current rows (lookups that must still pass freshness, such as a bank-rate table that can itself go stale) are judged by their declared policy, while plain reference rows are static. The public _trust_summary shows the distribution and explicitly counts missing freshness and row-count signals.

Discontinued — The source has stopped publishing new data. The data is frozen at the last known content date. This is not a freshness failure — it's a publisher decision.

Current distribution (_trust_summary): 137 fresh · 129 aging · 127 stale · 1 discontinued · 1 degraded · 3 unreachable · 6 unknown-freshness · 36 reference

Subscribe: RSS feed — get notified when dataset health changes.

Browser-dependent datasets

The current health summary identifies 0 browser-dependent sources (0.0% of the catalogue) that require a real browser to probe because their source pages render client-side JavaScript.

DataPulse uses Camofox, a self-hosted patched headless-Chromium sidecar, to probe these. The probe path is check.sh → Camofox sidecar → DOM snapshot → content-date extraction.

To enable browser probing:

1. Run the Camofox Docker sidecar on a reachable address (default http://localhost:9377). The probe script and the GitHub Actions workflow pick this up from the CAMOFOX_BASE_URL environment variable; nothing in this repo encodes a public IP.

2. Set CAMOFOX_BASE_URL to that address.

3. Restart the timer with systemctl restart datapulse-health.timer.

Without Camofox, these datasets will sit at browser-dependent — the