ossfind

# ossfind — safety-ranked open-source component discovery

Given a query like *"http client"*, ossfind returns open-source components **ranked by whether you
can actually ship a product on them** — a blended, explainable score of **fit · license · security ·
maintenance health · integration effort** — served through **both a web UI and an MCP tool** over one
ranking engine.

Its core promise: **never recommend ("ship") a component whose safety evidence is unsafe, missing, or
ambiguous.** The engine fails *closed*.

**New here?** → [`GETTING_STARTED.md`](GETTING_STARTED.md) — install, try the offline demo, go live,
and connect it to an AI agent over MCP (Claude Code / Claude Desktop / Cursor config included), in
about five minutes.

## Command line

```bash
npm install -g ossfind

ossfind search "markdown parser"
ossfind search "http client" -e cargo -n 5
ossfind search "web framework" -e pypi --json     # machine-readable
ossfind inspect marked                            # verified exports + import line
```

```
 1. marked  SHIP     92/100
    MIT  ·  0 CVEs  ·  OpenSSF 7.4
```

`-e/--ecosystem` npm · pypi · github · huggingface · cargo · rubygems · all — `-l/--license`
declares your project's license so incompatible results are ranked `AVOID` — `--json` for
scripting — `--no-color` (also honours `NO_COLOR`). Colour is disabled automatically when
stdout is not a terminal.

## Quick start (from source)

```bash
npm install
npm run typecheck && npm test     # 586 tests, fully offline
npm run gates                     # 17 safety gates, each proven able to fail
npm run eval                      # relevance against the labelled query set (live)
```

Run the web app (offline demo mode, uses frozen fixtures):

```bash
OSSFIND_FIXTURES=1 npm run web    # http://127.0.0.1:8787
```

By default, the web server binds exclusively to loopback (`127.0.0.1`).
- `HOST` — bind host (default `127.0.0.1`). Non-loopback hosts (e.g. `HOST=0.0.0.0`) require `OSSFIND_WEB_TOKEN` to be set; starting wide-open without a token is refused.
- `PORT` — server port (default `8787`).
- `OSSFIND_WEB_TOKEN` — optional Bearer token requiring `Authorization: Bearer <token>` on `/api/*` endpoints.

Run the MCP server (for AI agents):

```bash
OSSFIND_FIXTURES=1 npm run mcp    # stdio MCP server exposing `search_components`
```

Drop `OSSFIND_FIXTURES=1` to hit live suppliers (npm registry, ecosyste.ms, deps.dev, OSV).

## Ecosystems (npm · PyPI · crates.io · RubyGems · GitHub · Hugging Face)

ossfind searches **npm** (default), **PyPI**, **crates.io** (Rust), **RubyGems**, **GitHub** repositories, **Hugging Face** models, or
**all six at once** (`ecosystem: "all"`) — one query, results from every ecosystem merged and
safety-ranked together, so you don't have to guess where the answer lives (e.g. "video generation" →
PyPI's `decord`, a GitHub AI-model repo, and a Hugging Face model in the same result set). Pick the
ecosystem with the web/MCP selector, the `ecosystem` MCP tool argument, or `&ecosystem=all` on
`/api/search`.

Discovery is **federated**: a `FederatedDiscoverer` composes multiple source adapters per query
(parallel, per-source error isolation + timeouts, results merged and deduped by id). Enrichment routes
each candidate by its own id prefix (`npm:`/`pypi:`/`cargo:`/`rubygems:`/`github:`/`huggingface:`), so a mixed batch is
enriched correctly per-source. The safety-ranking layer is the same for every source — ossfind owns
the ranking, not the corpus. GitHub and Hugging Face are what surface AI-model repos/models (diffusers,
CogVideo, …) that aren't on any package registry.

- **npm** needs no key — discovery uses the npm registry search API, with query expansion
  (progressively shorter slices of the query, unioned) to recover the recall a conjunctive text
  match loses. Optionally federate it with a local semantic index to bridge vocabulary the
  registry cannot — `marked` says *parser* when you asked for a *renderer*:
  ```
  INDEX_MAX=8000 INDEX_DB_PATH=.cache/index/npm.db npm run index:build npm
  ```
  The same optional index federates crates.io, RubyGems and PyPI. When an index has not been
  built, that ecosystem's search behaves exactly as before.

  **It does not help everywhere, and the eval says where.** Measured on the labelled set:
  npm and crates.io improve substantially (crates.io MRR 0.000 → 0.675, since crates.io's own
  search ranks by name similarity and never returns `serde` for "serialization"). RubyGems is
  neutral on MRR and slightly positive on recall. Rebuilding PyPI's index concentrated on the
  top 8,000 packages measured slightly *worse* (0.611 → 0.597) than the broader 25,000-package
  index, so the wider corpus stays — a hypothesis the harness rejected.

  A RubyGems index originally measured much worse (MRR 0.500 → 0.250) by pushing `rails` out of
  the enrichment shortlist. That was a shortlisting defect, not an index one, and is fixed: a
  complete lexical match now counts as relevance evidence, so an adopted package whose
  description contains every query word earns a slot regardless of its embedding score.
  Measured on the labelled set, adding the index moved MRR 0.561 → 0.636, hit@3 60.5% → 67.4%
  and noise@3 2.6% → 0.0%, with no per-query regressions — and made `marked` the top result for
  *"markdown to html renderer"*, which no lexical probe can reach. Note that `npm run eval`
  therefore depends on a locally built index; without one the numbers are the registry-only ones.
- **GitHub** uses the repo search API. Set an optional `GITHUB_TOKEN` in `.env.local` for higher rate
  limits.
- **Hugging Face** needs no key — discovery uses the public models search API.
- **crates.io** (Rust) and **RubyGems** need no key — discovery uses their public search APIs,
  with licence/vulnerability/health enrichment from ecosyste.ms, OSV and deps.dev like any package
  ecosystem. crates.io ranks by name similarity, so "serialization" never returns `serde` from the
  registry alone; federate a local index (`INDEX_MAX=6000 INDEX_DB_PATH=.cache/index/cargo.db npm
  run index:build cargo`) to fix that.
- **Licence expressions**: an SPDX expression whose operands are *all* permissive resolves to a
  permissive licence — Rust's near-universal `MIT OR Apache-2.0` is a real choice, not an audit
  item. Any copyleft operand keeps the conservative treatment `G4` requires: `GPL-3.0 OR MIT`
  never ships into a permissive project, and a `WITH` exception or unrecognized operand is left
  for manual audit.
- **Health evidence is attributed only when the repository claim is corroborated.** A package's
  repository URL is self-declared, and typosquats name the real project's repo to inherit its
  OpenSSF score — five PyPI packages claiming `github.com/psf/requests` were reported SHIP 92/100
  on the real project's 8.1. The claim is now checked against the package name and fails closed
  (`G17`).
- **GitHub and Hugging Face components fail-closed to at most "caution"** (never "ship") — a raw repo's
  or model's dependency CVEs can't be verified the way a published package's can; Hugging Face also has
  no OpenSSF-style health score, so it relies on the existing missing-scorecard cap. License (SPDX) is
  still enriched and gated for both.
- **PyPI** discovery uses a **self-hosted local index** by default (no key, no third-party service).
  Build/refresh it once:
  ```
  INDEX_MAX=50000 npm run index:build          # top-N PyPI packages by downloads → .cache/index/pypi.db
  ```
  The index is `node:sqlite` FTS5 (BM25) over name/description/keywords, semantically reranked by the
  embedding model. Select the discovery source with `OSSFIND_PYPI_DISCOVERY=index|libraries|auto`
  (default `auto`: local index if built, else libraries.io).
- **libraries.io is the fallback** for PyPI (used when no local index exists). It needs a free key in
  a gitignored `.env.local` (`LIBRARY_IO_API_KEY=…`, `LIBRARIES_IO_API_KEY` also accepted), loaded via
  `node --env-file=.env.local …`. Without index or key, PyPI discovery degrades to empty (never crashes).

## Live mode & caching

Live mode stores successful supplier responses on disk to reduce repeat requests and avoid supplier
rate limits. Fixture mode remains local and does not use this cache.

- `OSSFIND_CACHE_DIR` — cache directory (default `.cache/http/`).
- `OSSFIND_CACHE_TTL` — cache lifetime in seconds for discovery, license, and health data (default `3600`).
- `OSSFIND_SECURITY_TTL` — cache lifetime in seconds for OSV vulnerability data (default `300`).
- `OSSFIND_CONCURRENCY` — maximum concurrent upstream enrichment requests (default `4`).
- `OSSFIND_NO_CACHE=1` — disable the live-response cache.

Security responses may be up to `OSSFIND_SECURITY_TTL` seconds stale; tune this value down when
stricter vulnerability-data freshness is required.

Supplier APIs are free but rate-limited; review each supplier's terms before commercial use.

## Telemetry & Usage Metrics

ossfind includes an in-memory, privacy-preserving usage collector that tracks aggregate operational health and supplier rate limits.

### Local Inspection (Read-Only)

You can inspect usage metrics at any time without sending data anywhere:
- **MCP Tool:** Call `usage_stats` to receive the metrics snapshot and a formatted summary of top suppliers, cache hit rates, rate-limit headroom, and latency percentiles (p50/p95).
- **Web API:** Send `GET /api/usage` to retrieve the JSON snapshot. When `OSSFIND_WEB_TOKEN` is set, `/api/usage` requires the same `Authorization: Bearer <token>` header as `/api/search`.

### What Is Collected
- **Aggregate Supplier Counters:** Total requests, cache hits, cache misses, HTTP status class counts (`2xx`, `4xx`, `5xx`), 429 counts, error counts, and latest rate-limit headroom (`remaining`, `limit`, `reset`, `retryAfter`) per approved supplier host.
- **Search Operations:** Total searches served, ecosystem distribution (`npm`, `pypi`, `github`, `huggingface`), verdict distribution (`ship`, `cau