Skill UI

A comprehensive framework for rigorously testing and benchmarking LLM agents. It moves beyond simple pass/fail checks to assess complex behaviors, reliability metrics, and capability consistency across multiple runs. Ideal for production monitoring and identifying subtle failure modes in advanced AI agents.

A comprehensive toolkit for evaluating the performance of code generation models. It supports benchmarking against industry standards like HumanEval, MBPP, and MultiPL-E across multiple languages. Use this tool to quantitatively compare different LLM coding abilities and measure code synthesis quality using pass@k metrics.

This skill is designed for running comprehensive evaluations of Hugging Face Hub models on local hardware. It supports multiple evaluation frameworks, including `inspect-ai` and `lighteval`, and provides flexibility in backend selection (e.g., `vllm`, Transformers, or `accelerate`) to optimize for performance and compatibility. Use this for local GPU benchmarking, smoke testing, and backend selection, but note that it does not handle remote job orchestration or result publishing.

A universal infrastructure for automatically evolving and optimizing AI agents across any domain. Using LLM-driven mutation and iterative solve-observe-evolve cycles, it enhances critical components like prompts, skills, and memory by benchmarking performance against measurable datasets, enabling the creation of self-improving AI systems.

Ruflo is a powerful, cross-agent orchestration layer designed to manage complex, multi-step AI coding workflows. It supports coordinating large swarms of AI agents, managing persistent memory through semantic search, and executing sophisticated pipelines using 30+ specialized plugins. Ideal for large-scale development tasks, system benchmarking, and deep research requiring coordinated agent effort.

This comprehensive guide instructs researchers on how to design rigorous and credible evaluations for Machine Learning systems. It covers critical areas such as workload selection, baseline comparison discipline, reporting four key coordinates (throughput, latency tails, memory, cost), structuring ablations, and generating scaling evidence. It ensures measurements meet academic benchmarking standards.

This skill provides an in-depth, entity-level scorecard for clinical or biomedical Named Entity Recognition (NER) models. It calculates precision, recall, and F1 scores (both strict and partial match modes) using a user-supplied gold corpus. Key features include per-label error breakdown, a confusion matrix, and detailed False Negative/False Positive analysis, essential for debugging model performance in highly sensitive medical domains.

Guides users to select a local, on-device Personally Identifiable Information (PII) detection model from a committed registry. It filters models based on language, required runtime environment (e.g., PyTorch, MLX), and size constraints. The process emphasizes building an offline shortlist and mandates rigorous pre-deployment recall validation and performance benchmarking to ensure secure, privacy-preserving deployment.

Generate decision-ready reports from AI agent evaluation runs. This skill ensures autonomous, assisted, failed, and timed-out outcomes remain distinct and comparable. It defines metrics with correct denominators, handles latency and cost populations honestly, and maps evidence to decision gates. Ideal for benchmarking, regression testing, and production readiness validation of AI agents.

This skill provides a standardized framework for evaluating autonomous AI agents. It tracks task completion rates, planning accuracy, tool invocation precision, and operational costs. Suitable for engineering teams managing distributed systems, it enables reproducible benchmarking, regression monitoring, and compliance tracking through structured data models like CSV, SQL, and JSON schemas.

Master the art of evaluating AI, LLM, and agent systems. Learn to decide when an evaluation is needed, design robust experiments, choose appropriate graders, and interpret results under hostile audit. Covers everything from spot checks to CI gates and benchmarking.

This tool evaluates Large Language Models across 60+ academic benchmarks like MMLU, GSM8K, and HumanEval. It is an industry standard used by major labs for benchmarking model quality, comparing models, and tracking training progress. Supports HuggingFace, vLLM, and APIs. Ideal for researchers and developers validating AI performance.