evidence-assessment

# Evidence Assessment

Valid reasoning from false premises proves nothing. Evidence assessment is the discipline of testing whether the premises of an argument are actually supported by the data, what kind of data would be decisive, and how much confidence the available evidence warrants. This skill covers the evaluation of sources, study designs, sampling, levels of evidence, and the integration of new evidence with prior knowledge.

**Agent affinity:** elder (structural reconstruction of evidence claims), tversky (base rates, inductive strength), kahneman-ct (evidence evaluation under System 1/2)

**Concept IDs:** crit-evidence-quality, crit-sourcing, crit-burden-of-proof, crit-scientific-literacy

## The Assessment Toolbox at a Glance

| # | Operation | Question answered |
|---|---|---|
| 1 | Source identification | Where did this claim come from? |
| 2 | Primary vs. secondary | Is this the original source or a report of it? |
| 3 | Source credibility | Does the source have relevant expertise and a track record? |
| 4 | Funding and conflicts | Who paid for this? What do they stand to gain? |
| 5 | Sample quality | How were the data collected and from whom? |
| 6 | Study design | Observational? Experimental? Randomized? |
| 7 | Levels of evidence | Where does this study sit in the evidence hierarchy? |
| 8 | Reproducibility | Has this result been replicated? |
| 9 | Scope check | What does the evidence actually show vs. what is being claimed? |
| 10 | Base rate integration | How does this update against prior probability? |

## The Levels of Evidence Hierarchy

Not all evidence is created equal. Roughly from weakest to strongest:

| Level | Type | Strength | Examples |
|---|---|---|---|
| 1 | Anecdote / testimonial | Very weak | "My cousin took X and felt better" |
| 2 | Expert opinion | Weak (unless grounded) | An authority says it's true |
| 3 | Case report | Weak | Single clinical observation |
| 4 | Case series | Weak | A handful of similar cases reported together |
| 5 | Cross-sectional / correlational study | Moderate | Survey data showing X and Y co-occur |
| 6 | Case-control study | Moderate | Retrospective comparison of cases to controls |
| 7 | Cohort study (prospective) | Moderate-strong | Follow a population forward over time |
| 8 | Randomized controlled trial (single) | Strong | Random assignment, controlled conditions |
| 9 | Meta-analysis / systematic review | Strongest | Pooled data from multiple RCTs with appropriate weighting |

**Key discipline.** The level of evidence should match the confidence of the claim. A single anecdote may be worth noting but cannot support "X causes Y." A meta-analysis can support stronger claims.

## Operation 1 — Source Identification

**Pattern:** Trace the claim to its origin. Who first said it? Where was it published? When?

**Worked example.** A tweet says, "Studies show that 8 glasses of water a day improves cognitive performance by 23%." The 23% is suspiciously precise. Trace it: the tweet cites a blog post, which cites a news article, which mentions "a study" without linking. Go find the actual study. Often, it does not exist, or the actual finding is much weaker ("effects on a specific subtest of working memory in a 15-person study").

**Discipline.** Refuse to cite or propagate a claim whose original source you have not seen.

## Operation 2 — Primary vs. Secondary Sources

**Primary source.** The original document — the research paper, the dataset, the eyewitness account, the legal ruling, the historical artifact.

**Secondary source.** A report, summary, interpretation, or citation of the primary source. Textbooks, news articles, review papers, and blog posts are usually secondary.

**Why it matters.** Secondary sources introduce errors. A claim filtered through three secondary sources typically loses nuance, acquires hedges or loses them, and drifts from the original. For high-stakes claims, go to the primary.

## Operation 3 — Source Credibility

Assess credibility along multiple dimensions:

- **Expertise.** Does the source have relevant training and experience in this specific area?
- **Track record.** Has the source been accurate in the past? Accountable for errors?
- **Institutional standing.** Is the source affiliated with institutions that impose quality controls (peer review, editorial standards, professional accountability)?
- **Incentives.** Does the source benefit from the claim being true?
- **Willingness to be wrong.** Does the source publicly update when evidence changes?

**Common mistake.** Treating credentials as a substitute for evaluation. An expert in one field is not automatically credible in another. A Nobel laureate in physics speaking about nutrition should be evaluated on the same terms as anyone else speaking about nutrition.

## Operation 4 — Funding and Conflicts of Interest

A source's funding or affiliations do not automatically invalidate their claims, but they do raise the bar for independent verification. Studies funded by industries with a stake in the outcome show systematically different results on average.

**Worked example.** A study funded by a food industry group finds no evidence linking its product to a health outcome. This does not prove the study wrong, but it raises the question of whether an independently funded replication would find the same result.

**Discipline.** Always note funding sources. Treat results from conflicted sources as weaker evidence than equivalent results from independent sources. Ask for replication in independent labs before accepting policy-relevant claims.

## Operation 5 — Sample Quality

For claims based on samples (surveys, studies, polls):

| Question | What to check |
|---|---|
| Sample size | Larger is better, with diminishing returns; 1,500 for population surveys is typical |
| Randomness | Was the sample drawn randomly from the target population? |
| Representativeness | Does the sample match the population in age, gender, region, etc.? |
| Self-se