Phoenix 大模型应用评估工具

v20260925

phoenix-evals

Phoenix Evals 是一款用于构建和运行 AI/LLM 应用评估器的工具。它支持代码评估和 LLM 评估，涵盖 RAG 系统、数据集创建、实验运行及生产环境监控。用户可通过 Python 或 TypeScript 实现模型性能验证，确保系统可靠性与准确性。

人工智能大模型评估 RAG Python TypeScript 验证实验

489 次下载

概览

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Task	Files
Setup	setup-python, setup-typescript
Decide what to evaluate	evaluators-overview
Choose a judge model	fundamentals-model-selection
Use pre-built evaluators	evaluators-pre-built
Build code evaluator	evaluators-code-python, evaluators-code-typescript
Build LLM evaluator	evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame	evaluate-dataframe-python
Understand experiments	experiments-overview
Run experiment	experiments-running-python, experiments-running-typescript
Create dataset	experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data	experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy	validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review	observe-sampling-python, observe-sampling-typescript
Analyze errors	error-analysis, error-analysis-multi-turn, axial-coding
RAG evals	evaluators-rag
Avoid common mistakes	common-mistakes-python, fundamentals-anti-patterns
Production	production-overview, production-guardrails, production-continuous

Starting Fresh:
observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator:
fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems:
evaluators-rag → evaluators-code- (retrieval) → evaluators-llm- (faithfulness)

Production:
production-overview → production-guardrails → production-continuous

Prefix	Description
`fundamentals-*`	Types, scores, anti-patterns
`observe-*`	Tracing, sampling
`error-analysis-*`	Finding failures
`axial-coding-*`	Categorizing failures
`evaluators-*`	Code, LLM, RAG evaluators
`experiments-*`	Datasets, running experiments
`validation-*`	Validating evaluator accuracy against human labels
`production-*`	CI/CD, monitoring

Principle	Action
Error analysis first	Can't automate what you haven't observed
Custom > generic	Build from your failures
Code first	Deterministic before LLM
Validate judges	>80% TPR/TNR
Binary > Likert	Pass/fail, not 1-5

信息

分类人工智能

名称 phoenix-evals

版本 v20260925

大小 43.97KB

来源 github/awesome-copilot

更新时间 2026-09-27