Skill UI

A comprehensive workflow for implementing robust LLM evaluation using Langfuse. This skill guides users through key processes including scoring traces (numeric, categorical, boolean), collecting user feedback, managing versioned prompts, creating structured datasets, running automated experiments, and implementing LLM-as-a-Judge patterns for advanced quality assurance and system benchmarking.

Master comprehensive strategies for evaluating Large Language Model (LLM) applications. This skill covers automated metrics (BLEU, ROUGE, BERTScore), structured human evaluation dimensions, and advanced LLM-as-Judge techniques. Use it to systematically measure model performance, compare prompts, detect regressions, and build confidence in production systems.

This skill guides users on setting up and running LLM-as-judge evaluations using Arize. It covers defining evaluators (including prompts, classification choices, and models) and executing tasks against real data spans or experiments. It supports key metrics like hallucination, faithfulness, and correctness, and facilitates continuous monitoring for comprehensive LLM performance analysis.

This harness provides comprehensive, reproducible evaluation pipelines for complex LLM chains and agents (LangChain/LangGraph 1.0). It integrates golden dataset management, LangSmith evaluation, RAGAS metrics, deepeval LLM-as-judge, and structured agent trajectory analysis. Use it when establishing quality benchmarks for new chains, diagnosing performance regressions after model switches, or implementing CI/CD gates to prevent quality drops.

This tool leverages an LLM-as-a-Judge agent to thoroughly evaluate staged Git changes before they are committed. It checks for correctness, completeness, and safety issues, providing a detailed verdict (APPROVE, NEEDS_REVIEW, REJECT) and actionable feedback. It is essential for maintaining high code quality, especially in complex or security-sensitive code sections.

This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It includes designing rubrics, direct scoring, and pairwise comparisons to mitigate biases like position and length bias. Suitable for building reliable evaluation systems, debugging inconsistent results, and establishing quality standards across teams.

This skill enables the creation of robust evaluation frameworks for AI agent systems. It is used for systematically testing agent performance, validating context engineering choices, and measuring improvements over time. The approach accounts for non-determinism and multiple valid paths, utilizing multi-dimensional rubrics, LLM-as-judge methods, and human evaluation to ensure quality gates and catch regressions before deployment.

Master comprehensive evaluation strategies for LLM applications, covering automated metrics like BLEU and ROUGE, human evaluation dimensions, and LLM-as-Judge patterns. This skill helps measure performance, compare models, detect regressions, and validate prompt improvements. Includes Python implementation examples for metrics and evaluation suites.

A public-safe preliminary screen for challenges to the impartiality of an AI evaluator, LLM-as-a-judge, or decision-support agent based on prior outputs, suspected bias, unstable reasoning, or self-reinforcing memory. Classifies evidence and recommends proportionate remedies.

Builds the Phase 0 evaluation harness that gates every fine-tuning run: golden sets from trace error analysis, per-failure-mode graders, calibrated LLM judges, drift suite, and base-model baseline.

Design trustworthy LLM and agent evaluations: weighted rubrics, LLM-as-judge calibration, inter-rater agreement, and Elo/Bradley-Terry pairwise ranking of model or prompt variants.