Skill UI

A comprehensive framework for rigorously testing and benchmarking LLM agents. It moves beyond simple pass/fail checks to assess complex behaviors, reliability metrics, and capability consistency across multiple runs. Ideal for production monitoring and identifying subtle failure modes in advanced AI agents.

A comprehensive toolkit for evaluating the performance of code generation models. It supports benchmarking against industry standards like HumanEval, MBPP, and MultiPL-E across multiple languages. Use this tool to quantitatively compare different LLM coding abilities and measure code synthesis quality using pass@k metrics.

A comprehensive workflow for implementing robust LLM evaluation using Langfuse. This skill guides users through key processes including scoring traces (numeric, categorical, boolean), collecting user feedback, managing versioned prompts, creating structured datasets, running automated experiments, and implementing LLM-as-a-Judge patterns for advanced quality assurance and system benchmarking.

This skill is designed for running comprehensive evaluations of Hugging Face Hub models on local hardware. It supports multiple evaluation frameworks, including `inspect-ai` and `lighteval`, and provides flexibility in backend selection (e.g., `vllm`, Transformers, or `accelerate`) to optimize for performance and compatibility. Use this for local GPU benchmarking, smoke testing, and backend selection, but note that it does not handle remote job orchestration or result publishing.

A universal infrastructure for automatically evolving and optimizing AI agents across any domain. Using LLM-driven mutation and iterative solve-observe-evolve cycles, it enhances critical components like prompts, skills, and memory by benchmarking performance against measurable datasets, enabling the creation of self-improving AI systems.

Arbor executes an Autonomous Optimization (AO) loop, allowing users to iteratively improve complex artifacts—such as models, agents, or data pipelines—against a measurable objective. It implements Hypothesis Tree Refinement (HTR), structuring research into a persistent tree to track hypotheses, insights, and evidence. This process is crucial for long-horizon optimization tasks (e.g., Kaggle-style benchmarking) and utilizes a held-out test set to prevent overfitting and ensure cumulative, auditable progress.

This skill executes the DeepSWE benchmark, a comprehensive coding-agent evaluation suite. It uses OpenRouter and mini-swe-agent to score various Large Language Models (LLMs) on 113 tasks, providing reproducible results for model comparison, development assessment, and research benchmarking. Use this when a standardized, deep evaluation of LLM coding capabilities is required.

This skill provides a detailed, side-by-side comparison of advanced LLM agent evaluation frameworks, such as ruflo and HAL. It helps researchers and developers identify capability gaps—including real code execution, full web browsing, and structured file parsing—and prioritize architectural improvements for robust agent benchmarking and performance tracking.

This tool executes the advanced `metaharness/darwin` security benchmark (ADR-155). It rigorously evaluates the security detection capabilities of AI systems against a controlled ground-truth corpus, providing critical metrics like TPR, FPR, and patch-pass rates. It compares the champion model against multiple baselines, offering empirical data crucial for iterative security improvement and drift detection.

Ruflo is a powerful, cross-agent orchestration layer designed to manage complex, multi-step AI coding workflows. It supports coordinating large swarms of AI agents, managing persistent memory through semantic search, and executing sophisticated pipelines using 30+ specialized plugins. Ideal for large-scale development tasks, system benchmarking, and deep research requiring coordinated agent effort.

A comprehensive guide for positioning and refining computer vision manuscripts for the IEEE/CVF International Conference on Computer Vision (ICCV). This skill helps authors assess conference fit, structure evidence, perform benchmarking, and adopt the necessary narrative for a flagship CV submission, covering topics from foundation models to video recognition.

This comprehensive guide instructs researchers on how to design rigorous and credible evaluations for Machine Learning systems. It covers critical areas such as workload selection, baseline comparison discipline, reporting four key coordinates (throughput, latency tails, memory, cost), structuring ablations, and generating scaling evidence. It ensures measurements meet academic benchmarking standards.