dspy-embedding-retrieval
This skill should be used when the user asks to "build local DSPy retrieval", "use dspy.Embedder", "use dspy.Embeddings", "save an embeddings index", "add FAISS retrieval", mentions semantic search, hosted embeddings, local embedding models, `EmbeddingsWithScores`, or needs a DSPy retriever over an application-owned text corpus.
git clone --depth 1 https://github.com/OmidZamani/dspy-skills /tmp/dspy-embedding-retrieval && cp -r /tmp/dspy-embedding-retrieval/skills/dspy-embedding-retrieval ~/.claude/skills/dspy-embedding-retrievalSKILL.md
# DSPy Embedding Retrieval
## Goal
Build semantic retrieval over an application-owned text corpus with `dspy.Embedder` and `dspy.Embeddings`.
## Basic Hosted Embedder
```python
import dspy
corpus = [
"DSPy programs are composed from modules.",
"MIPROv2 optimizes instructions and demonstrations.",
"RLM explores large contexts with a sandboxed REPL.",
]
embedder = dspy.Embedder("openai/text-embedding-3-small")
search = dspy.Embeddings(corpus=corpus, embedder=embedder, k=2)
result = search("Which optimizer tunes prompts?")
print(result.passages)
print(result.indices)
```
## Use in RAG
```python
class LocalRAG(dspy.Module):
def __init__(self, retriever):
super().__init__()
self.retriever = retriever
self.answer = dspy.ChainOfThought("context: list[str], question -> answer")
def forward(self, question: str):
context = self.retriever(question).passages
return self.answer(context=context, question=question)
```
## Custom Local Embeddings
Wrap any callable that accepts `list[str]` and returns a 2D numeric array:
```python
from sentence_transformers import SentenceTransformer
import dspy
model = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1")
embedder = dspy.Embedder(model.encode)
search = dspy.Embeddings(corpus=corpus, embedder=embedder, k=5)
```
## Scores, FAISS, and Persistence
Use `dspy.EmbeddingsWithScores` when downstream logic needs similarity thresholds or reranking.
For corpora at or above the `brute_force_threshold` default of `20_000`, DSPy builds a FAISS index. Install FAISS first:
```bash
pip install faiss-cpu
```
Persist the index when embedding the corpus is expensive:
```python
search.save("./retrieval-index")
loaded = dspy.Embeddings.from_saved("./retrieval-index", embedder=embedder)
```
## Related Skills
- Build a complete pipeline: [dspy-rag-pipeline](../dspy-rag-pipeline/SKILL.md)
- Design typed context fields: [dspy-signature-designer](../dspy-signature-designer/SKILL.md)
- Harden caches: [dspy-production-deployment](../dspy-production-deployment/SKILL.md)
## Best Practices
1. Evaluate retrieval quality separately from answer quality.
2. Keep corpus chunking deterministic and versioned.
3. Persist expensive indexes.
4. Use `EmbeddingsWithScores` when debugging relevance.
5. Measure memory and latency before enabling FAISS for large corpora.
## Official Documentation
- **Embedder API**: https://dspy.ai/api/models/Embedder/
- **Embeddings API**: https://dspy.ai/api/tools/Embeddings/Use this skill when you need to QA audit and fix a plugin skill file. Provides a methodology for verifying skill content against official documentation, fixing issues in-place, and producing verification reports.
This skill should be used when the user asks to "choose a DSPy adapter", "use JSONAdapter", "use XMLAdapter", "enable native function calling", "send images, audio, or files to DSPy", mentions `dspy.ChatAdapter`, `dspy.JSONAdapter`, `dspy.XMLAdapter`, `dspy.Image`, `dspy.Audio`, `dspy.File`, structured outputs, or multimodal DSPy signatures.
This skill should be used when the user asks to "compose DSPy modules", "use Ensemble optimizer", "combine multiple programs", "use dspy.MultiChainComparison", mentions "ensemble voting", "module composition", "sequential pipelines", or needs to build complex multi-module DSPy programs with ensemble patterns or multi-chain comparison.
This skill should be used when the user asks to "use BetterTogether", "combine prompt optimization and fine-tuning", "sequence DSPy optimizers", "run prompt then weight optimization", mentions `dspy.BetterTogether`, strategy strings such as "p -> w -> p", or needs to compose multiple DSPy teleprompters into an evaluated optimization sequence.
This skill should be used when the user asks to "bootstrap few-shot examples", "generate demonstrations", "use BootstrapFewShot", "optimize with limited data", "create training demos automatically", mentions "teacher model for few-shot", "10-50 training examples", or wants automatic demonstration generation for a DSPy program without extensive compute.
This skill should be used when the user asks to "create custom DSPy module", "design a DSPy module", "extend dspy.Module", "build reusable DSPy component", mentions "custom module patterns", "module serialization", "stateful modules", "module testing", or needs to design production-quality custom DSPy modules with proper architecture, state management, and testing.
This skill should be used when the user asks to "debug DSPy programs", "trace LLM calls", "monitor production DSPy", "use MLflow with DSPy", mentions "inspect_history", "custom callbacks", "observability", "production monitoring", "cost tracking", or needs to debug, trace, and monitor DSPy applications in development and production.
This skill should be used when the user asks to "evaluate a DSPy program", "test my DSPy module", "measure performance", "create evaluation metrics", "use answer_exact_match or SemanticF1", mentions "Evaluate class", "comparing programs", "establishing baselines", or needs to systematically test and measure DSPy program quality with custom or built-in metrics.