paper-preflight

# paper-preflight

<!-- mcp-name: io.github.amos689/paper-preflight -->

**English** · [简体中文](README.zh-CN.md)

[![CI](https://github.com/amos689/paper-preflight/actions/workflows/ci.yml/badge.svg)](https://github.com/amos689/paper-preflight/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
![Python 3.11–3.14](https://img.shields.io/badge/python-3.11%E2%80%933.14-blue.svg)

**Check every reference of a LaTeX paper against real scholarly records before you submit.
No LLM guessing, no false accusations.**

![paper-preflight checking the demo paper: errors for an undefined citation key, a DOI that belongs to another paper, a reference no source knows and a retracted paper; warnings for a published preprint, a wrong year and a LaTeX-escaped DOI](https://raw.githubusercontent.com/amos689/paper-preflight/main/docs/demo/demo.gif)

Language models invent references, and copy-pasted BibTeX carries wrong years, wrong authors
and dead DOIs. paper-preflight reads your `.tex` and `.bib` files and asks Crossref, dblp,
arXiv, DataCite, PubMed and OpenAlex (and Semantic Scholar, if you have a key) about every cited
work:

- Does it exist?
- Does it match what you wrote?
- Has it been retracted?
- Has the preprint you cite been published since?

When it cannot tell, it says so instead of guessing.

> **Status: v0.4, an early release.** False positives are the bugs we most want to hear
> about: please [open an issue](https://github.com/amos689/paper-preflight/issues).

The repository's [demo paper](examples/demo-paper) cites eleven works, several of them wrong on
purpose. A real run, against the live sources:

```text
$ paper-preflight check examples/demo-paper
paper-preflight 0.4.0 · main.tex · 12 entries, 12 cited keys

error   CIT001 main.tex:31
    Citation key 'nonexistent2023' is not defined in any bibliography file (1 use(s)).
error   REF001 refs.bib:43
    The doi of 'devlin2019bert' (10.1109/cvpr.2016.90) resolves to a different work in Crossref: "Deep Residual Learning for Image Recognition" (He et al., 2016).
error   REF003 refs.bib:66
    'lindqvist2024quantum' was not found in Crossref, dblp and Semantic Scholar, and every source responded. Check that the work exists and that its title is correct.
error   REF004 refs.bib:73
    'wakefield1998ileal' has been retracted (reported by Crossref, OpenAlex). Cite it only if the text discusses the retraction.
error   CIT002 refs.bib:127
    Entry key 'kingma2015adam' is already defined at line 47; BibTeX ignores this one.
warning REF015 refs.bib:31
    'he2015residual' cites a preprint that has been published in CVPR (2016), DOI 10.1109/cvpr.2016.90. Cite the published version and keep the eprint field.
warning CIT004 refs.bib:37
    Entries 'devlin2019bert' and 'he2016deep' look like the same work (same DOI).
warning REF013 refs.bib:51
    'kingma2015adam' gives the year 2016, but dblp records 2014, 2015.
warning REF017 refs.bib:111
    The doi of 'tacl2019example' contains LaTeX escapes: '10.1162/tacl\_a\_00276'. Write it as: 10.1162/tacl_a_00276
info    REF005 refs.bib:73
    'wakefield1998ileal' has a published correction (reported by Crossref).
info    REF090 refs.bib:86
    'goodfellow2016deep' could not be verified: grey literature without an identifier (book, report, software, web page).
info    REF090 refs.bib:94
    'zhou2016ml' could not be verified: non-Latin titles are not supported yet; grey literature without an identifier (book, report, software, web page).
info    CIT003 refs.bib:115
    Entry 'lecun1998gradient' is never cited.

References: 6 verified · 1 metadata mismatch · 1 identifier conflict · 1 not found · 2 cannot determine
5 error(s) · 4 warning(s) · 4 info
```

Each finding is backed by a record (or by every source answering "no"). The correct NeurIPS
paper is verified through dblp even though Crossref only holds fake copies of it, and the two
books without identifiers are reported as "cannot determine" instead of "not found".

## What it catches

| Rule | Finding |
|---|---|
| REF001 | The DOI or arXiv ID points to a different paper |
| REF002 | The DOI or arXiv ID does not exist |
| REF003 | The work was not found in any source, and every source answered |
| REF004 · REF005 | The work was retracted, or has an expression of concern or a correction |
| REF010–REF014 | Authors, title, year or venue differ from the real record |
| REF015 | A cited preprint has been formally published |
| REF016 | The registry has a DOI the entry lacks (offered as a safe fix) |
| REF017 | An identifier is written so that links break (`10.1162/tacl\_a\_00276`, `…v1`) |
| CIT001–CIT008 | Undefined, duplicate, unused or near-duplicate citation keys; broken `.bib` syntax |
| REF090 | Cannot determine, always with the reason (source unavailable, grey literature, …) |

`paper-preflight explain REF003` describes any rule.

## How accurate is it?

Four measurements, all against the live sources: hallucinations found in published papers, the
bibliographies of real papers, a head-to-head with published tools, and a public benchmark.

### On hallucinations that got past peer review

GPTZero published 151 hallucinated references it found in NeurIPS 2025 papers and ICLR 2026
submissions, each confirmed by its staff. Pasted as plain text, as the papers printed them:

| References | Flagged | Cannot determine | Verified |
|---|---|---|---|
| 151 | **135 (89%)** | 16 | **0** |

- **None of them is verified.** The 16 left undecided are web pages and blog posts, titles too
  short to search with confidence, real titles given with invented authors where several works
  share the title, and two references the plain-text reader could not take apart. Each is listed
  with its reason in [`evals/results/gptzero.md`](evals/results/gptzero.md).
- GPTZero's own tool found these, so they are the hallucinations a search can find; recall on
  every kind of hallucination is lower (see HALLMARK below).

### On real papers

The bibliographies of 20 arXiv papers first submitted in mid-August 2026 (cs, stat, q-bio,
quant-ph and astro-ph), chosen mechanically and collected only after every fix in this release,
with every warning and error reviewed by hand:

| References | Flags | Real problems | False positives | Unclear | False positives per 100 references |
|---|---|---|---|---|---|
| 753 | 86 | 77 | 9 | 0 | 1.2 |

- **One false alarm every two papers** (38 references on average), against 77 real problems:
  36 errors in the entries (invented authors and titles, wrong given names, years and titles,
  identifiers written so that links break) and 41 cited preprints that have since been
  published.
- **The false alarms are mostly registry records with errors of their own** (two misspelt
  titles, an affiliation mark inside a name, a workshop filed under a joint volume) **and real
  works no source describes as cited** (a Substack post, a technical report, a database cited
  by its access year, an article's early-access year).
- **Six earlier batches of 20 papers were used to find false positives,** each first measured
  as it came out (0.1.0: 4.5 per 100 references; 0.1.1: 2.3; 0.1.2 before its last fixes: 3.0;
  0.1.2: 1.7; 0.2.1: 1.9; 0.3.0: 1.9). Details in [`evals/README.md`](evals/README.md#real-papers).

### Next to other tools

[Badalova & Mayr (2026)](https://doi.org/10.5281/zenodo.21457492) checked 104 references by hand
and published what five tools flagged. On the same references, with their labels:

| Tool | Precision [95% CI] | Recall | False flags per 100 correct references |
|---|---|---|---|
| CheckIfExist | 47.7% [36.0%, 59.6%] | 93.9% | 47.9 |
| HalluCiteChecker | 47.4% [32.5%, 62.7%] | 54.5% | 28.2 |
| Hallucinator | 50.9% [38.3%, 63.4%] | 87.9% | 39.4 |
| HalRef | 31.2% [21.9%, 42.2%] | 72.7% | 74.6 |
| RefChecker | 47.1% [35.7%, 58.8%] | 97.0% | 50.7 |
| **paper-preflight** | **72.5% [57.2%, 83.9%]** | 87.9% | **15.5** |

The sample is small, so the intervals are wide. Some flags count as false here because the study
labels a reference correct when the work exists: five of paper-preflight's flags on such
references point at real errors (a wrong author, a broken DOI). Two causes of false flags found
in this data were fixed, and four names the study's CSV garbled were restored, before the run
above; the first run measured 62.8%. See
[`evals/results/badalova-mayr.md`](evals/results/badalova-mayr.md).

### On a benchmark: HALLMARK

[HALLMARK](https://github.com/rpatrik96/hallmark) is a public benchmark of real and hallucinated
BibTeX entries.

| Split | Mode | Precision | Recall | False-positive rate | Coverage |
|---|---|---|---|---|---|
| `test_public`: 831 entries, never used during development | Any issue | 98.1% | 88.9% | 2.2% | 97.0% |
| | Fabrication | 99.0% | 49.0% | 0.6% | 97.0% |
| `dev_public`: 1,119 entries, used during development | Any issue | 97.6% | 90.7% | 2.1% | 98.4% |
| | Fabrication | 98.1% | 52.7% | 1.0% | 98.4% |

HALLMARK v1.2.3, every entry of both public splits, run on 2026-10-04. *Fabrication* counts a
wrong identifier, a work not found and no author in common; *any issue* also counts wrong
authors, title, year or venue.

- **The held-out split confirms the development numbers:** the same precision and two points
  less recall on entries no rule was ever tuned on.
- **Every flag on a `dev_public` entry labelled VALID was checked by hand.** The 11 that remain are not
  correct citations: DOIs that belong to other papers, author lists naming people who did not
  write the paper, a shifted year and a truncated title.
- **Without them, both modes reach 100% precision and 0% false positives.** The list, each item
  with a reason one lookup confirms, is in
  [`evals/hallmark_disputed.toml`](evals/hallmark_disputed.toml).
- **What is still missed:** invented venues on papers known only as preprints (an arXiv record
  cannot contradict a venue) and author lists that merely leave people out. See
  [`evals/results/