Skill UI

A comprehensive framework for rigorously testing and benchmarking LLM agents. It moves beyond simple pass/fail checks to assess complex behaviors, reliability metrics, and capability consistency across multiple runs. Ideal for production monitoring and identifying subtle failure modes in advanced AI agents.

A comprehensive toolkit for evaluating the performance of code generation models. It supports benchmarking against industry standards like HumanEval, MBPP, and MultiPL-E across multiple languages. Use this tool to quantitatively compare different LLM coding abilities and measure code synthesis quality using pass@k metrics.

Evaluates large language models across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) with standardized prompts and metrics, making it easy to benchmark HuggingFace or API models, compare releases, and track training progress.

Track and improve production LLMs with Langfuse observability: trace API calls, manage prompts, score evaluations, integrate with LangChain/LlamaIndex/OpenAI, and keep dashboards on cost, latency, and quality.

A comprehensive workflow for implementing robust LLM evaluation using Langfuse. This skill guides users through key processes including scoring traces (numeric, categorical, boolean), collecting user feedback, managing versioned prompts, creating structured datasets, running automated experiments, and implementing LLM-as-a-Judge patterns for advanced quality assurance and system benchmarking.

LangSmith centralizes tracing, evaluation, and monitoring for LLM applications, letting teams debug prompts, run regression tests, collect latency/cost metrics, and integrate with OpenAI, Anthropic, LangChain, or LlamaIndex to keep production systems observable.

Master comprehensive strategies for evaluating Large Language Model (LLM) applications. This skill covers automated metrics (BLEU, ROUGE, BERTScore), structured human evaluation dimensions, and advanced LLM-as-Judge techniques. Use it to systematically measure model performance, compare prompts, detect regressions, and build confidence in production systems.

Enterprise NeMo Evaluator SDK runs over 100 benchmarks from 18+ harnesses with multi-backend execution (Docker, Slurm, cloud) to give reproducible, scalable LLM evaluation workflows.

Phoenix is an open-source observability platform for tracing, evaluating, and monitoring LLM applications; use it to debug prompts, run dataset evaluations, build experiment pipelines, and watch production inference in real time.

This skill specializes in designing, optimizing, and evaluating prompts for Large Language Models (LLMs). It allows users to generate optimized prompt templates, structured output schemas (e.g., JSON mode, function calling), and comprehensive test suites. Use this skill when building new LLM applications, implementing complex reasoning patterns like Chain-of-Thought, defining system guardrails, or building robust evaluation frameworks to measure and improve model performance.

This comprehensive workflow guides the end-to-end implementation of Retrieval-Augmented Generation (RAG) systems. It covers critical stages, including selecting optimal embedding models, setting up vector databases, designing chunking strategies, implementing hybrid retrieval, integrating Large Language Models (LLMs), and performing rigorous evaluation. Use this when building knowledge-grounded AI applications or advanced semantic search systems.

A comprehensive, model-agnostic toolkit for advanced LLM development. It enables iterative prompt optimization through static analysis (token usage, clarity), measures critical RAG metrics (relevance, faithfulness, coverage), and validates complex agent workflows and configurations. Essential for rigorous LLM quality assurance and cost management.