qikly

# qikly

<!-- The MCP Registry proves that whoever lists a server owns the package it
     points at, by looking for this line in the README that PyPI serves. It is
     a comment so it does not render, and it must match the `name` in
     server.json exactly. -->
<!-- mcp-name: io.github.gal-a/qikly -->

[![tests](https://github.com/gal-a/qikly/actions/workflows/ci.yml/badge.svg?branch=main&event=push)](https://github.com/gal-a/qikly/actions/workflows/ci.yml)
[![pypi](https://img.shields.io/pypi/v/qikly?color=blue)](https://pypi.org/project/qikly/)
[![python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](https://pypi.org/project/qikly/)
[![license](https://img.shields.io/badge/license-Apache%202.0-blue)](https://github.com/gal-a/qikly/blob/main/LICENSE)
[![marketplace](https://img.shields.io/badge/GitHub%20Marketplace-Qikly%20Test%20Generation-2b8f95)](https://github.com/marketplace/actions/qikly-test-generation)
[![VS Code](https://img.shields.io/badge/VS_Code-Install_qikly_MCP-0098FF?logo=visualstudiocode&logoColor=white)](https://vscode.dev/redirect/mcp/install?name=qikly&config=%7B%22name%22%3A%22qikly%22%2C%22command%22%3A%22qikly-mcp%22%7D)

**The problem: Your AI writes both the code and its tests. How do you know the tests are really valid?**

**The solution: two agents.** One turns the acceptance criteria into tests.
The other writes the code and **never sees the acceptance criteria.**

**Who it is for:** a developer or team pointing an AI coding agent at a
self-contained Python module that transforms data, for example an ETL step, a
merge, a calculation or a validation routine, who does not want to trust a
green suite when the same agent wrote both the code and the tests. It suits one
module at a time: when a test fails, only the files that failure names are
loaded, so runs stay small and quick. Large multi-file repositories are a
different problem. See [What it is for](#what-it-is-for).

Imagine a student who writes the exam paper, writes the answer key, and then
sits the exam. They pass. Obviously they pass, and nobody would accept that as
evidence the student knows the material.

That is what happens when one model is given a specification containing the
acceptance criteria and asked to produce both the implementation and the suite
that checks it. It writes tests its own code will pass. Everything goes green,
and the green means nothing.

qikly takes the answer key away from the student.

It generates a test suite from the acceptance criteria in a specification,
then writes an implementation and repairs it against that suite until every
test passes or a retry budget runs out. The agent doing the writing and
repairing never sees the criteria. Every failure, every piece of
reasoning and every diff is recorded.

**The part that makes the result mean something:** the coding agent never sees
`acceptance_criteria`. It gets the specification with that section stripped
out, the same vague brief a developer works from, while test generation gets
it in full. When a test fails, the agent sees the failure message and never
the rule it broke. Without that asymmetry both sides read the same spec
identically and every test passes first try, which proves nothing.

```mermaid
flowchart TD
    SPEC["<b>Full specification</b><br/>task.yaml<br/>requirements + interface<br/>acceptance_criteria"]
    REQ["requirements<br/>+ interface"]
    AC["acceptance_criteria"]
    CODE["<b>Coding agent</b><br/>writes the implementation<br/>FIX then PATCH on failure"]
    TEST["<b>Test-writing agent</b><br/>writes the suite"]
    IMPL["Implementation"]
    SUITE["<b>pytest suite</b><br/>Tests for:<br/>1 integration, 2 system,<br/>then 3 unit"]
    RUN{"Run the suite"}
    FAIL["<b>Failure errors</b> only<br/>no criteria, no test source"]
    OUT["Converged<br/><b>outputs:</b> code + suite<br/>+ audit trail"]
    STALL["Did not converge<br/><b>failure errors and audit trail</b><br/>exits non-zero, ships nothing"]

    SPEC --> REQ
    SPEC --> AC
    AC -. "never reaches" .-x CODE
    REQ --> CODE
    REQ --> TEST
    AC --> TEST
    CODE --> IMPL
    TEST --> SUITE
    IMPL --> RUN
    SUITE --> RUN
    RUN -- pass --> OUT
    RUN -- fail --> FAIL
    FAIL -- "repair loop:<br/>FIX, then PATCH" --> CODE
    FAIL -- "retry budget spent" --> STALL
    IMPL -. "unit stage only:<br/>written last, from the code" .-> TEST

    classDef codeView fill:#f3e8ff,stroke:#7e22ce,color:#4c1d95
    classDef standardView fill:#d9ebea,stroke:#0e6a70,color:#0b3d40
    classDef converged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef stalled fill:#fdf0d5,stroke:#b45309,color:#78350f
    class CODE,IMPL,FAIL codeView
    class AC,TEST,SUITE standardView
    class OUT converged
    class STALL stalled
    linkStyle 10 stroke:#15803d,stroke-width:2px
    linkStyle 11,12 stroke:#7e22ce,stroke-width:2px
    linkStyle 13 stroke:#b45309,stroke-width:2px
```

**Purple is what the coding agent can see. Teal is what the standard is
written from.** They never touch. A run that never converges is still worth having: it exits
non-zero, names the blocking tests, and keeps the same complete record. The purple arrows are the repair loop, and that is where
almost all of a run happens: a failing suite sends the agent the failure text
and nothing else, it produces a FIX and a PATCH, and the suite runs again,
until the stage passes or the retry budget runs out. It never sees the rule it
broke, so it cannot write code shaped to a criterion it was shown.
`tests/test_withholding.py` fails the build if any call site lets one through.

A run works through three stages, `integration` then `system` then `unit`:

1. **Integration and system tests are generated first**, from the spec alone,
   before any code exists. They cannot see an implementation because there is
   not one yet.
2. **The coding agent writes the implementation**, from the spec minus the
   criteria.
3. **pytest runs.** On failure the model produces a **FIX** (failure summary,
   root cause, plan, and the files it intends to touch) and then a **PATCH**
   (a unified diff of only those files), applied all or nothing. Repeat until
   the stage passes or the budget is spent.
4. **Unit tests are generated last**, once real code exists for them to name.
   This is the only stage allowed to see the implementation.
5. **Clearing a stage re-runs the earlier ones**, so a later fix cannot
   silently break something that already passed.


Every arrow back into **FIX** carries the pytest error text and nothing else.
Unit tests come last because they are the only ones that need to name real
functions, which makes them the only stage allowed to read the implementation.
Re-running the earlier stages after each success is what stops a later repair
quietly breaking something that already passed.

### Two ways to get a test suite, and what each can prove

| | Code-derived suite<br>*most commercial test generators, and qikly's own unit tests* | Spec-derived suite<br>*qikly's integration and system tests* |
|---|---|---|
| **Written from** | The code as it is today | The acceptance criteria you wrote |
| **You supply** | Nothing but the repository | A written statement of what correct means |
| **Catches** | Behaviour changing tomorrow | Behaviour being wrong today |
| **Cannot catch** | The code being wrong now: today's bug becomes tomorrow's assertion | Anything nobody wrote down |
| **A green run means** | The code still does what it did when the tests were generated | The code satisfies the criteria as written |
| **A red run means** | Someone changed behaviour, deliberately or not | The code and your stated intent disagree |
| **Fails you when** | The original behaviour was already wrong | The criteria are vague, missing, or wrong |
| **Right choice when** | Nobody wrote the intent down and you need a safety net | The intent exists in a ticket, a spec page or a Gherkin file |

Both are useful and they answer different questions. qikly is not purely one
or the other: integration and system tests are written from the criteria
before any code exists, the unit stage is written last from the code that just
passed them, and `--refine-criteria` reads a converged implementation to
propose criteria the first draft missed. Each of those reads the code on
purpose, and none of them can question it.

## What makes this different

**The tests come from the standard, not from the code.** This is the one
that matters most. Every other AI test generator in this space writes its tests
*from an implementation that already exists*, so it can only describe what the
code already does. That makes an excellent regression harness, and it cannot
tell you the code is wrong. qikly writes the integration and system suites from
the acceptance criteria **before any implementation exists**, so the standard
cannot have been shaped by the thing it judges.

**The withholding is a mechanism you can watch.** Not a prompt asking a
model to ignore a section, and not a convention someone has to remember. One
command prints what each side is given and the difference between them,
offline and free:

```bash
qikly --explain <MY_TASK>     # e.g. qikly --explain MERGE_SALES
```

Eleven criteria go to test generation. Twelve lines are removed before the
coding agent sees the same file. `tests/test_withholding.py` fails the build if
any call site ever lets one through, including one added next year by someone
who has never read this. It is a property of the code, and it takes thirty
seconds to check.

**What you get is an executable suite you keep.** The output is pytest files
and JUnit XML. Read them, run them, put them in CI, and when one fails in six
months it fails for a reason you can inspect and argue with. A suite is a
durable asset in a way a model's verdict is not: a verdict cannot be re-run
against tomorrow's commit.

**It helps you write the standard, not just check against it.** `--init` and
`--scaffold` turn