arize-phoenix

# Arize Phoenix

Phoenix is an open-source AI observability platform built on OpenTelemetry that helps developers understand, debug, and improve AI applications. It provides comprehensive tracing, evaluation, prompt engineering, and experimentation capabilities for LLM-based systems. Phoenix captures detailed execution information from AI applications, measures output quality with evaluators, enables systematic prompt iteration, and supports data-driven experimentation to optimize AI performance.

## When to Use This Skill

- Debugging AI application failures by inspecting LLM calls, tool executions, and retrieval operations
- Measuring and improving AI output quality using LLM-based or code-based evaluators
- Iterating on prompts using real production examples and testing variations systematically
- Comparing different versions of AI applications (prompts, models, architectures) using experiments
- Monitoring LLM costs, token usage, latency, and error rates in production
- Building datasets from production traces for evaluation and fine-tuning
- Tracking multi-turn conversations and maintaining context across interactions
- Optimizing RAG systems by analyzing retrieval quality and document relevance
- Evaluating agent performance including tool call accuracy and actionability
- Managing prompt versions and deploying them across different environments

## Capabilities

Agents can leverage Phoenix to:

- **Trace** AI application execution with detailed visibility into LLM calls, tool executions, retrieval operations, embeddings, and prompt templates
- **Evaluate** output quality using pre-built or custom evaluators with LLM-as-a-judge or code-based evaluation logic
- **Annotate** traces with human feedback, scores, labels, and quality signals for continuous improvement
- **Experiment** systematically by comparing different versions of applications using datasets and evaluators
- **Monitor** performance metrics including latency, token usage, costs, and error rates across projects
- **Iterate** on prompts using the playground, span replay, and dataset-based testing
- **Organize** traces into projects and sessions for better management and analysis
- **Integrate** with 20+ AI frameworks and LLM providers via OpenTelemetry instrumentation

## Skills

### Tracing

- **Capture traces** via OpenTelemetry (OTLP) protocol with automatic instrumentation for major frameworks
- **View execution flow** showing every LLM call, tool execution, retrieval operation, embedding generation, and response generation
- **Inspect LLM parameters** including temperature, system prompts, function calls, and invocation parameters
- **Analyze retrieval operations** with document scores, order, and embedding text for RAG systems
- **Track token usage** with detailed breakdowns by token type (input/output) and model
- **Monitor latency** at trace, span, and component levels with quantile analysis
- **Organize with projects** to separate traces by environment, application, team, or use case
- **Group with sessions** to track multi-turn conversations and maintain context across interactions
- **Add metadata** to traces with custom attributes, tags, and structured data for filtering and analysis
- **Annotate traces** with scores, labels, human feedback, and LLM evaluations for quality measurement
- **Export and import traces** for backup, migration, or analysis in external tools
- **Track costs** with automatic calculation based on token usage and model pricing

### Evaluation

- **Run LLM-as-a-judge evaluations** using any LLM provider (OpenAI, Anthropic, Gemini, custom endpoints) to assess output quality
- **Build custom evaluators** with Python or TypeScript using custom prompts, scoring logic, and evaluation criteria
- **Use pre-built evaluators** for common tasks including faithfulness, relevance, toxicity, summarization, agent evaluation, and RAG quality
- **Write code-based evaluators** for deterministic checks like exact match, regex patterns, or custom Python/TypeScript logic
- **Execute evaluations at scale** with automatic concurrency, rate limit handling, error management, and batching via executors
- **Map complex inputs** using input schemas and mappings to transform nested data structures for evaluators
- **View evaluator traces** with complete transparency into prompts, model reasoning, scores, and execution metadata
- **Run batch evaluations** on traces, datasets, or custom data sources with automatic retry and error handling
- **Integrate evaluations** into workflows by running evals on production traces or test datasets

### Datasets & Experiments

- **Create datasets** from traces, code, CSV files, or manually curated examples with inputs and optional reference outputs
- **Build golden datasets** with reference outputs (ground truth) for objective evaluation using code-based evaluators
- **Version datasets** with automatic tracking of inserts, updates, and deletes for reproducibility
- **Run experiments** by executing task functions against datasets with evaluators to compare different versions
- **Compare experiments** side-by-side in the UI to see performance differences, score distributions, and individual example results
- **Use repetitions** to run experiments multiple times for statistical confidence and account for LLM variability
- **Organize with splits** to separate datasets into train/test/validation splits for proper evaluation workflows
- **Export datasets** in JSONL or CSV formats for fine-tuning, analysis, or sharing
- **View experiment results** in the Phoenix UI with task function traces, scores per example, and aggregate performance metrics

### Prompt Engineering

- **Manage prompts** with versioning, storage, and deployment across different environments
- **Test prompts interactively** in the Prompt Playground with various models, parameters, and tools
- **Replay LLM spans** from production traces in the playground to debug failures and test improvements
- **Test at scale** by run