Skill UI

A comprehensive tool for designing, generating schemas, and evaluating complex multi-agent systems. It guides users through defining system requirements, selecting optimal orchestration patterns (Supervisor, Swarm, Pipeline), generating provider-ready tool schemas (Anthropic/OpenAI), and analyzing execution logs for performance bottlenecks, cost, and latency. This ensures robust, production-grade autonomous workflows.

A comprehensive framework for rigorously testing and benchmarking LLM agents. It moves beyond simple pass/fail checks to assess complex behaviors, reliability metrics, and capability consistency across multiple runs. Ideal for production monitoring and identifying subtle failure modes in advanced AI agents.

Provides comprehensive guidance on building robust evaluation frameworks for complex agent systems. It addresses the inherent challenges of non-determinism and multiple valid paths, enabling systematic performance testing, validating context engineering choices, and measuring continuous improvements across multiple quality dimensions (e.g., factual accuracy, tool efficiency).

A comprehensive, model-agnostic toolkit for advanced LLM development. It enables iterative prompt optimization through static analysis (token usage, clarity), measures critical RAG metrics (relevance, faithfulness, coverage), and validates complex agent workflows and configurations. Essential for rigorous LLM quality assurance and cost management.

Scaffolds production-ready generative AI agents on Vertex AI with Gemini models, multi-modal tools, RAG, and Cloud Run/Endpoint deployment, including observability and evaluation for enterprise scenarios.

This comprehensive guide outlines a rigorous, human-in-the-loop engineering workflow for leveraging advanced AI agents to complete complex implementation tasks. It mandates defining clear completion criteria, decomposing work into manageable units, and strategically routing models (Haiku, Sonnet, Opus) based on task complexity. The workflow emphasizes continuous evaluation, regression testing, security review of invariants and edge cases, and strict cost discipline to ensure robust and reliable software development using AI assistance.

This pattern defines a robust, continuous loop structure for autonomous agents. It integrates crucial components like quality gates, evaluation harnesses, and explicit recovery controls. It guides complex workflows by defining decision paths (e.g., PR control, RFC decomposition) and includes strategies to mitigate common failure modes such as infinite loop churn or cost drift, ensuring measurable progress in autonomous systems.

Use this command to set up a structured multi-agent collaboration environment within AgentHub. It allows you to define the core task, specify the required number of agents, and configure detailed evaluation criteria (metrics, directions, and commands) for both quantitative benchmarking and qualitative LLM judging. This is the starting point for any agent competition or complex AI workflow.

Autonomous experiment loop that optimizes any file by a measurable metric. The agent edits a target file, runs a fixed evaluation, keeps improvements (git commit), discards failures (git reset), and loops indefinitely. Ideal for optimizing code speed, reducing bundle size, improving test pass rates, enhancing prompts, or boosting content quality through iterative measured attempts.

A skill for orchestrating parallel AI subagents that compete on the same task using isolated git worktrees. Handles session setup, agent spawning, progress monitoring, evaluation via metrics or LLM judge, and final merge of the winning branch. Ideal for code optimization, content variation, research exploration, and any task benefiting from parallel competition.

This tool allows users to systematically evaluate and rank the outputs of multiple agents within an AgentHub session. It supports metric-based evaluation (running custom commands for measurable metrics like speed or resource usage), or an advanced LLM Judge Mode. The LLM Judge compares results based on criteria like correctness, simplicity, and overall quality, helping users determine the best-performing agent for the task.

A lightweight CLI tool for systematically comparing various coding AI agents (such as Claude Code, Aider, and Codex) head-to-head. It measures crucial metrics—pass rate, API cost, execution time, and consistency—on reproducible tasks defined via YAML manifests. Use this to objectively select the best agent for your codebase or to validate model updates before production deployment.