Skill UI

A comprehensive guide to building production-grade evaluation systems for Large Language Models (LLMs). This skill covers advanced methodologies, including Direct Scoring and Pairwise Comparison, alongside critical techniques for mitigating systematic biases (e.g., Position Bias, Length Bias). Learn how to select appropriate metrics and structure prompts to ensure reliable, consistent, and objective quality assessment across various AI applications.

This skill provides comprehensive tools for documenting and managing model performance on Hugging Face. It supports multiple methods for adding structured evaluation data, including extracting tables from README content, importing benchmark scores via APIs (like Artificial Analysis), and executing high-performance custom evaluations using advanced backends such as vLLM or lighteval. It ensures that all metadata is structured and compatible with model leaderboards and industry standards.

This skill is designed for running comprehensive evaluations of Hugging Face Hub models on local hardware. It supports multiple evaluation frameworks, including `inspect-ai` and `lighteval`, and provides flexibility in backend selection (e.g., `vllm`, Transformers, or `accelerate`) to optimize for performance and compatibility. Use this for local GPU benchmarking, smoke testing, and backend selection, but note that it does not handle remote job orchestration or result publishing.

This skill provides comprehensive management capabilities for AI Integrations within the Arize platform. Users can create, list, retrieve, and update credentials for various Large Language Model (LLM) providers, including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, and Vertex AI. These integrations are crucial for enabling Arize's advanced monitoring and evaluation features that require invoking external LLMs.

This skill guides users on setting up and running LLM-as-judge evaluations using Arize. It covers defining evaluators (including prompts, classification choices, and models) and executing tasks against real data spans or experiments. It supports key metrics like hallucination, faithfulness, and correctness, and facilitates continuous monitoring for comprehensive LLM performance analysis.

This skill provides a comprehensive framework for optimizing, improving, and debugging Large Language Model (LLM) prompts. It leverages production trace data, system evaluations, and human annotations to extract prompts from spans. Users can run a data-driven optimization loop, ensuring the AI responses meet desired quality standards. Ideal for prompt tuning, system prompt refinement, and improving overall AI output quality.

A comprehensive evaluation suite designed to benchmark the performance of Large Language Models (LLM) and Vision-Language Models (VLM) specifically for home security applications. It covers 143 diverse tests across 16 critical domains, including tool use, prompt injection resistance, contextual reasoning, alert triage, and detailed scene analysis, ensuring robustness for smart home AI systems.

This harness provides comprehensive, reproducible evaluation pipelines for complex LLM chains and agents (LangChain/LangGraph 1.0). It integrates golden dataset management, LangSmith evaluation, RAGAS metrics, deepeval LLM-as-judge, and structured agent trajectory analysis. Use it when establishing quality benchmarks for new chains, diagnosing performance regressions after model switches, or implementing CI/CD gates to prevent quality drops.

This skill executes the DeepSWE benchmark, a comprehensive coding-agent evaluation suite. It uses OpenRouter and mini-swe-agent to score various Large Language Models (LLMs) on 113 tasks, providing reproducible results for model comparison, development assessment, and research benchmarking. Use this when a standardized, deep evaluation of LLM coding capabilities is required.