Skill UI

A comprehensive framework for rigorously testing and benchmarking LLM agents. It moves beyond simple pass/fail checks to assess complex behaviors, reliability metrics, and capability consistency across multiple runs. Ideal for production monitoring and identifying subtle failure modes in advanced AI agents.

A comprehensive toolkit for evaluating the performance of code generation models. It supports benchmarking against industry standards like HumanEval, MBPP, and MultiPL-E across multiple languages. Use this tool to quantitatively compare different LLM coding abilities and measure code synthesis quality using pass@k metrics.

A comprehensive workflow for implementing robust LLM evaluation using Langfuse. This skill guides users through key processes including scoring traces (numeric, categorical, boolean), collecting user feedback, managing versioned prompts, creating structured datasets, running automated experiments, and implementing LLM-as-a-Judge patterns for advanced quality assurance and system benchmarking.

This skill is designed for running comprehensive evaluations of Hugging Face Hub models on local hardware. It supports multiple evaluation frameworks, including `inspect-ai` and `lighteval`, and provides flexibility in backend selection (e.g., `vllm`, Transformers, or `accelerate`) to optimize for performance and compatibility. Use this for local GPU benchmarking, smoke testing, and backend selection, but note that it does not handle remote job orchestration or result publishing.

A universal infrastructure for automatically evolving and optimizing AI agents across any domain. Using LLM-driven mutation and iterative solve-observe-evolve cycles, it enhances critical components like prompts, skills, and memory by benchmarking performance against measurable datasets, enabling the creation of self-improving AI systems.

This skill executes the DeepSWE benchmark, a comprehensive coding-agent evaluation suite. It uses OpenRouter and mini-swe-agent to score various Large Language Models (LLMs) on 113 tasks, providing reproducible results for model comparison, development assessment, and research benchmarking. Use this when a standardized, deep evaluation of LLM coding capabilities is required.

This skill provides a detailed, side-by-side comparison of advanced LLM agent evaluation frameworks, such as ruflo and HAL. It helps researchers and developers identify capability gaps—including real code execution, full web browsing, and structured file parsing—and prioritize architectural improvements for robust agent benchmarking and performance tracking.

Ruflo is a powerful, cross-agent orchestration layer designed to manage complex, multi-step AI coding workflows. It supports coordinating large swarms of AI agents, managing persistent memory through semantic search, and executing sophisticated pipelines using 30+ specialized plugins. Ideal for large-scale development tasks, system benchmarking, and deep research requiring coordinated agent effort.

A comprehensive guide for positioning and refining computer vision manuscripts for the IEEE/CVF International Conference on Computer Vision (ICCV). This skill helps authors assess conference fit, structure evidence, perform benchmarking, and adopt the necessary narrative for a flagship CV submission, covering topics from foundation models to video recognition.

This comprehensive guide instructs researchers on how to design rigorous and credible evaluations for Machine Learning systems. It covers critical areas such as workload selection, baseline comparison discipline, reporting four key coordinates (throughput, latency tails, memory, cost), structuring ablations, and generating scaling evidence. It ensures measurements meet academic benchmarking standards.

This skill provides an in-depth, entity-level scorecard for clinical or biomedical Named Entity Recognition (NER) models. It calculates precision, recall, and F1 scores (both strict and partial match modes) using a user-supplied gold corpus. Key features include per-label error breakdown, a confusion matrix, and detailed False Negative/False Positive analysis, essential for debugging model performance in highly sensitive medical domains.

Guides users to select a local, on-device Personally Identifiable Information (PII) detection model from a committed registry. It filters models based on language, required runtime environment (e.g., PyTorch, MLX), and size constraints. The process emphasizes building an offline shortlist and mandates rigorous pre-deployment recall validation and performance benchmarking to ensure secure, privacy-preserving deployment.