x-archive-rag

# x-archive-rag

Turn your X/Twitter archive into a local AI memory layer.

[![Release](https://img.shields.io/github/v/release/mameshivaa/x-archive-rag?label=release)](https://github.com/mameshivaa/x-archive-rag/releases)
[![CI](https://github.com/mameshivaa/x-archive-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/mameshivaa/x-archive-rag/actions)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![GitHub stars](https://img.shields.io/github/stars/mameshivaa/x-archive-rag?style=social)](https://github.com/mameshivaa/x-archive-rag/stargazers)

Unlike uploading a ZIP to ChatGPT, `x-archive-rag` converts your archive into reusable local infrastructure: SQLite storage, repeatable retrieval, citation IDs, persona profiling, grounded draft prompts, a local web UI, and MCP tools that AI clients can query without receiving your whole archive.

The default workflow keeps your archive local. You decide when to export data or send a small retrieved evidence set to an OpenAI-compatible model.

If this is useful for your personal AI or MCP workflow, starring the repository helps other people find it: <https://github.com/mameshivaa/x-archive-rag>

## What is technically different?

`x-archive-rag` is not a bigger prompt. It is a small local data layer for your archive.

| Capability | Why it matters |
| --- | --- |
| Local SQLite index | Import once, query repeatedly, inspect or back up the database. |
| Keyword + semantic + hybrid retrieval | Ask narrow questions without sending the whole archive every time. |
| Citation-grounded outputs | Search and draft prompts preserve tweet IDs such as `T1`. |
| Persona profile as separate state | Writing patterns are computed once and reused with retrieved evidence. |
| CLI, local web UI, export, and MCP | The same archive can serve humans, scripts, and AI clients. |
| Read-only MCP tools | Claude or another MCP client can retrieve evidence without posting to X or calling a remote model. |

## Local web UI

![Local web UI showing archive stats, search, and draft prompt controls](docs/assets/web-ui.png)

Run it locally after importing an archive:

```bash
x-archive-rag web --db ./memory.sqlite --open
```

Try the UI with the included sample archive before touching private data:

```bash
git clone https://github.com/mameshivaa/x-archive-rag.git
cd x-archive-rag
python3 -m pip install -e .
x-archive-rag demo --open
```

## Terminal demo

![Terminal demo showing ingest, search, draft, and MCP usage](docs/assets/terminal-demo.svg)

## What you can do

1. Import your official X archive ZIP or extracted directory into a local SQLite database.
2. Search it with keyword, semantic, or hybrid retrieval.
3. Generate a persona profile from your own posts.
4. Build citation-grounded draft prompts that show the tweets used as evidence.
5. Expose the same local memory to AI clients through MCP tools.

## Interface

The main interface is a small CLI with nine commands:

- `ingest` turns an archive into a reusable local database.
- `search` retrieves relevant posts with citation IDs.
- `profile` summarizes recurring topics, tone, and writing patterns.
- `draft` builds a grounded prompt from retrieved evidence.
- `mcp` exposes the same memory to AI clients that support MCP.
- `web` runs a local browser UI for search, profile inspection, and draft prompts.
- `demo` creates a sample database and can open the local web UI.
- `doctor` checks whether a database is ready for search, draft, and MCP use.
- `export` writes imported tweets as JSONL or Markdown for inspection and reuse.

Text output is designed for humans. JSON and Markdown output are designed for scripts, agents, and repeatable workflows.

## Status

Alpha. The current release is intentionally small:

- Import `tweets.js` / `tweet.js` from an extracted archive directory or archive ZIP, including common long-form tweet text fields and expanded URL entities.
- Store normalized tweets in SQLite.
- Search with SQLite FTS5, dependency-free semantic retrieval, or hybrid retrieval.
- Generate a persona profile from your own posts.
- Build grounded RAG prompts with cited tweets.
- Expose archive search and draft prompts through a minimal MCP stdio server.
- Optionally call OpenAI-compatible chat completions using `OPENAI_API_KEY`.

## Why

Most archive tools help you search old posts. This project focuses on a different workflow:

> "Given what I have said before, draft something that sounds like me, while showing the evidence."

The assistant is designed for drafting and reflection, not impersonation or automated posting.

## Why not just upload the ZIP to ChatGPT?

Uploading a ZIP is a one-off context dump. `x-archive-rag` is a reusable memory layer.

| ZIP upload | `x-archive-rag` |
| --- | --- |
| Sends the archive to a remote chat session. | Imports the archive into a local SQLite database. |
| Relies on one model pass over a large attachment. | Retrieves only relevant tweets per question. |
| Hard to reproduce or inspect later. | Keeps repeatable search, export, and doctor checks. |
| Evidence can disappear into a summary. | Preserves citation IDs such as `T1` in search and draft output. |
| Works only inside that chat session. | Exposes CLI, JSON, Markdown export, and MCP tools. |

For a deeper technical comparison, see [docs/comparison.md](docs/comparison.md).

## Install

From a local checkout:

```bash
python3 -m pip install -e .
```

No runtime dependencies are required.

## Quick start

Want to test without private archive data first?

```bash
x-archive-rag demo --open
```

The shortest path is:

```bash
x-archive-rag ingest ./twitter-archive.zip --db ./memory.sqlite
x-archive-rag profile --db ./memory.sqlite
x-archive-rag search "personal AI memory" --db ./memory.sqlite --mode hybrid
x-archive-rag draft "What do I think about OSS personal AI?" --db ./memory.sqlite
x-archive-rag web --db ./memory.sqlite --open
```

Use the sections below when you need more control over import format, date filters, exports, OpenAI-compatible calls, or MCP.

Want the Claude Desktop Extension artifact instead? Download the latest `.mcpb` from [Releases](https://github.com/mameshivaa/x-archive-rag/releases/latest) and follow [docs/mcpb.md](docs/mcpb.md).

AI coding agents and MCP marketplace reviewers can start from [llms.txt](llms.txt) or [llms-install.md](llms-install.md).

Prefer a browser surface? Run the local web UI and follow [docs/web.md](docs/web.md).

Import an extracted X archive directory:

```bash
x-archive-rag ingest ./twitter-archive --db ./x-archive-rag.sqlite
```

Retweets are excluded by default so profiles and drafts focus on your own statements. Add `--include-retweets` if you want to index retweets too.

Or import the original ZIP:

```bash
x-archive-rag ingest ./twitter-archive.zip --db ./x-archive-rag.sqlite
```

Search your archive:

```bash
x-archive-rag search "AI agents and personal tools" --db ./x-archive-rag.sqlite
```

Search and draft outputs include citation IDs such as `T1`, so generated text can refer back to specific retrieved posts.

Example search output:

```text
[T1] 2024-01-01T12:00:00+00:00 https://x.com/i/web/status/1001
Personal AI should be local-first and cite the memories it uses.
```

JSON output is available for scripts:

```bash
x-archive-rag search "AI agents and personal tools" \
  --db ./x-archive-rag.sqlite \
  --json
```

Semantic and hybrid search are available without external services:

```bash
x-archive-rag search "grounded memory for drafts" \
  --db ./x-archive-rag.sqlite \
  --mode hybrid
```

Limit retrieval to a time range when older views should not influence a draft:

```bash
x-archive-rag search "personal AI" \
  --db ./x-archive-rag.sqlite \
  --since 2024-01-01 \
  --until 2024-12-31
```

Build a persona profile:

```bash
x-archive-rag profile --db ./x-archive-rag.sqlite
```

Profiles include the number of tweets analyzed and the covered date range.

Draft with retrieved context:

```bash
x-archive-rag draft "What do I think about OSS personal AI?" \
  --db ./x-archive-rag.sqlite \
  --mode hybrid \
  --output ./draft.md
```

The draft command builds an evidence-grounded prompt. It does not call a remote model unless `--call-openai` is set.

```text
Retrieved tweets:
[T1] 2024-01-01T12:00:00+00:00 https://x.com/i/web/status/1001
Personal AI should be local-first and cite the memories it uses.

Task:
Draft a response that is grounded in the retrieved tweets and consistent with the persona profile.
```

Call an OpenAI-compatible model:

```bash
export OPENAI_API_KEY="..."
x-archive-rag draft "What do I think about OSS personal AI?" \
  --db ./x-archive-rag.sqlite \
  --call-openai \
  --model gpt-4.1-mini
```

Use an OpenAI-compatible endpoint:

```bash
x-archive-rag draft "What do I think about local-first AI?" \
  --db ./x-archive-rag.sqlite \
  --call-openai \
  --model local-model \
  --base-url http://localhost:8000/v1
```

Run as an MCP stdio server:

```bash
x-archive-rag mcp --db ./x-archive-rag.sqlite
```

Available MCP tools:

- `search_archive`
- `draft_from_archive`
- `get_persona_profile`

Check database readiness:

```bash
x-archive-rag doctor --db ./x-archive-rag.sqlite
```

Export imported tweets for inspection or backup:

```bash
x-archive-rag export \
  --db ./x-archive-rag.sqlite \
  --format jsonl \
  --output ./tweets.jsonl
```

See [docs/cli.md](docs/cli.md) for the full CLI reference.

## Design

The pipeline is deliberately simple:

```text
X archive ZIP or directory
  -> tweet normalization
  -> persistent SQLite memory + FTS5 + lightweight semantic vectors
  -> per-prompt retrieval with citation IDs
  -> persona profile
  -> grounded draft prompt, optional LLM call, or MCP tool response
```

Future releases can add external embedding providers, thread reconstruction, and local model providers without changing the core import/search model. The current release already includes CLI, MCP, export, and local web UI surfaces over the