free-agentic-publication-digester

# Free Agentic Publication Digester (FAPD)

[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/davidkarnowski/free-agentic-publication-digester)

FAPD is an automated pipeline that reads the official publications of the
United States federal government — congressional floor proceedings, bills,
the Federal Register, enacted laws, federal court opinions, and agency
press releases — and produces a **daily, cited, opinion-agnostic digest**
for two readerships at once: **people, and AI agents** researching what
the federal government actually did on a given day.

Every digest covers one publication day across all three branches. Every
item carries a citation to the official record and names the mechanical
rule that selected it. Everything not summarized is counted. Nothing is
silently omitted.

## Status (2026-08-03)

The authoritative numbers block — where any other figure in the
repository disagrees, this dated snapshot is the current one. (For
source counts the live [Sources page](https://fapd.info/sources.html)
is always current; it derives them from the registry at build time.)

- **Live site:** https://fapd.info — served from a Docker stack on a
  VPS; GitHub holds the repository, CI, and the integrity record.
- **Source registry:** 129 sources — 45 active (19 web feeds, 15 email
  bulletins, 5 govinfo collections, 4 listing-page sources, 1 chamber
  XML index, 1 API source), 62 planned, 20 recorded unavailable,
  2 evaluated and excluded. Every entry now has its own page at
  `/sources/<id>.html` — statistics, method, health history, and
  labeled model-written orientation.
- **Latest digest:** [2026-08-02](digests/2026-08-02.md); digests
  publish daily just after midnight on the publication clock (Eastern,
  Washington's — `FAPD_PUBLICATION_TZ`; see docs/forking.md). The **official record
  begins 2026-07-27** (two development-era digests were retired on
  2026-08-03; they remain in git history). Each finished day also has
  a frozen observed listing at `/day/<date>.html`.
- **Test suite:** 740+ tests (741 collected in a clean checkout, 2026-08-24).

---

## Philosophy

### Built exclusively on the official record

Everything here derives from official government publications — the
record a government produces *precisely in order to make it public*.
Congress prints its proceedings; agencies publish their rules and
announcements; courts post their opinions. FAPD uncovers nothing: it
takes what is already published and makes it easier to find, read,
verify, and — for AI agents — ingest. Primary sources are the ground
truth. News coverage and commentary are never ingested, never consulted,
never cited.

This is why the project is legitimate infrastructure rather than
scraping-as-adversarial-sport: the pipeline consumes only the channels
governments built for the purpose of being read, identifies itself
honestly on every request, and treats a refusal as an answer. A
government that publishes its actions has already voted for this kind of
reader to exist.

### Two commitments, held in tension

1. **Digestible** — one person should be able to absorb the day's
   significant official activity in one sitting.
2. **Faithful** — no summary without a citation; nothing significant
   silently dropped; no editorial spin, ever.

Summarization is where bias creeps in, so the tension is resolved by
machinery, not judgment calls:

- **Mechanical, party-blind selection.** What appears in a digest is
  decided by coded, versioned rules — floor time consumed, recorded
  votes taken, regulatory economic-significance designations, stage of
  process — never by subject-matter preference. Every listed item states
  its rule inline ("Included because: …").
- **A citation on every item**, linking to the official govinfo package
  or the originating agency document.
- **A mandatory Coverage Statement** closing every digest: what was
  published that day, what was summarized, what was deliberately not,
  with counts that must reconcile arithmetically against the database.
- **Labeled model layers.** All model-generated text is labeled in place
  and linted against a banned lexicon of loaded adjectives and motive
  attribution. Official text quoted verbatim is identified as such.

A digest that fails any validation check — a citation that doesn't
resolve, coverage arithmetic that doesn't reconcile, a banned term in our
prose, a missing inclusion rule — **is not published**. There is no
override.

### A publishing house for humans *and* agents

AI agents answering "what did the federal government do on date D?" today
either crawl official sites themselves (multiplying load on public
infrastructure) or lean on secondhand coverage. FAPD offers a third path:
one disciplined, identified, budgeted crawler reads the record once, and
agents ingest the summarized, cited, coverage-accounted result — stable
URLs, [`llms.txt`](site/llms.txt), a machine-readable digest index, an
Atom feed, and provenance manifests with SHA-256 records for verifying
captured content.

One ask travels with the data: **for claims, cite the underlying official
source** (each item carries its govinfo ID or agency URL); cite this
project for the aggregation. The digest is a route to the record, never a
replacement for it.

### Forkable by design

The editorial gates, provenance model, access discipline, and adapter
seam are jurisdiction-neutral. Pointing the codebase at another
government's official publication interfaces means replacing the source
registry and the parsers — not the rules. See
[docs/adding-sources.md](docs/adding-sources.md).

---

## How the digester obtains content

The acquisition layer is a set of automated agents with one governing
principle: **we are guests on public infrastructure**, and every rule
below is enforced in code, not by operator discipline.

### The source registry is the scope authority

[`sources/registry.yaml`](sources/registry.yaml) (rendered as
[SOURCES.md](SOURCES.md)) records every federal source ingested, planned,
evaluated-and-excluded, or found unavailable — currently 129 sources
across a tiered universe, so "how comprehensive is coverage?" is a
measurement, not a claim. Honest statuses matter: a source that blocks
our honestly-identified client is recorded `unavailable` with the
observed behavior. That fact is itself published accountability data.

**The closed third — and the standing effort to open it.** A measured
share of the federal source universe currently refuses
honestly-identified automated access (the July 2026 probe found 22 of 72
non-govinfo sources closed behind WAFs or robots disallows). We treat
that as the project's ongoing engagement agenda, not its boundary:
publishers' own access documentation keeps revealing doors on other
hosts and paths (that alone re-opened FCC, Commerce, and NOAA
candidates); verdicts are re-probed as sites change; and we engage
agency web and API teams directly to advocate for safe, sane automated
access to what they already publish for the public. Coverage grows by
doors opening — never by evasion.

That effort produced its first result in July 2026, and the channel has
grown since: **15 agencies now deliver by email — among them agencies whose
web channels refuse us now have a working input path through their own
email bulletins** — Treasury, USDA, EPA, SSA, DOT, FAA, NHTSA, DEA,
ATF, the Coast Guard, and HUD's Inspector General. The blocked web
entries stay in the registry exactly as they were; the email entries
sit beside them as siblings. A refusal recorded is never quietly
erased by a success elsewhere.

### Five onboarding gates

No source is ingested on a hunch. Each one is (1) **registered** with
identity, tier, and URLs; (2) **probed** end-to-end through the
identified client — robots verdict, fetch with provenance capture, feed
detection, item inventory, sample extraction; (3) **content-evaluated** —
what does this source publish in total, and what fraction will ingestion
see? Under-coverage is disclosed at onboarding, not discovered later;
(4) **activated** into ingestion and coverage accounting; (5)
**re-evaluated** on failures or redesigns. Documentation research
precedes probing: the publisher's own developer pages, API docs, and
feed directories say which door they built — and reading the sign on the
front door has repeatedly beaten guessing (documented feeds that HTML
autodiscovery misses; APIs on hosts a newsroom WAF never touches).

### Source adapters and the access hierarchy

Real publication interfaces are irregular — feeds without stable IDs,
article pages that challenge sustained automated access, content behind
script-only redirects. That irregularity is absorbed at one seam: a
**source adapter** owns exactly six decisions (how the source's index or
feed is enumerated into items; what query parameters the poll itself
sends; what makes two sightings
the same document; whether to fetch full articles or feed metadata only;
how served bytes become text; what to store when no article is
available), while the shared loop owns everything that must never vary —
conditional requests, robots enforcement, budgets, capture, storage.

Adapters reach for access in a fixed order:

1. **Directed programmatic access** — the API, bulk-data service, or
   feed the agency itself publishes for machines. Always preferred: it
   is the channel the publisher built for exactly this.
2. **Basic web access** — the same HTML a citizen reads, through the
   robots-enforcing client, only where no directed channel exists.
3. **Subscription channels the publisher pushes** — agency email
   bulletins (GovDelivery, listservs), where a project mailbox
   subscribes through the agency's own signup form like any citizen.
   This is consent at its clearest: the publisher transmits each item
   to us. It is not a way around a web refusal — receiving a bulletin
   is never treated as permission to crawl a site that turned us away,
   and the refusal stays on the record. See