Skill UI

A comprehensive framework for rigorously testing and benchmarking LLM agents. It moves beyond simple pass/fail checks to assess complex behaviors, reliability metrics, and capability consistency across multiple runs. Ideal for production monitoring and identifying subtle failure modes in advanced AI agents.

A comprehensive toolkit for evaluating the performance of code generation models. It supports benchmarking against industry standards like HumanEval, MBPP, and MultiPL-E across multiple languages. Use this tool to quantitatively compare different LLM coding abilities and measure code synthesis quality using pass@k metrics.

A comprehensive workflow for implementing robust LLM evaluation using Langfuse. This skill guides users through key processes including scoring traces (numeric, categorical, boolean), collecting user feedback, managing versioned prompts, creating structured datasets, running automated experiments, and implementing LLM-as-a-Judge patterns for advanced quality assurance and system benchmarking.

This skill is designed for running comprehensive evaluations of Hugging Face Hub models on local hardware. It supports multiple evaluation frameworks, including `inspect-ai` and `lighteval`, and provides flexibility in backend selection (e.g., `vllm`, Transformers, or `accelerate`) to optimize for performance and compatibility. Use this for local GPU benchmarking, smoke testing, and backend selection, but note that it does not handle remote job orchestration or result publishing.

A robust, production-tested pattern for coordinating multiple specialized AI agents. The orchestrator decomposes complex tasks, routes them to the most suitable specialist, prevents redundant or conflicting work (anti-duplication), and enforces mandatory quality gates (e.g., running tests, checking file diffs) to verify results before marking a task as complete. It also includes heartbeat monitoring for task progress.

A universal infrastructure for automatically evolving and optimizing AI agents across any domain. Using LLM-driven mutation and iterative solve-observe-evolve cycles, it enhances critical components like prompts, skills, and memory by benchmarking performance against measurable datasets, enabling the creation of self-improving AI systems.

This tool executes the advanced `metaharness/darwin` security benchmark (ADR-155). It rigorously evaluates the security detection capabilities of AI systems against a controlled ground-truth corpus, providing critical metrics like TPR, FPR, and patch-pass rates. It compares the champion model against multiple baselines, offering empirical data crucial for iterative security improvement and drift detection.