Skill UI

A comprehensive framework for rigorously testing and benchmarking LLM agents. It moves beyond simple pass/fail checks to assess complex behaviors, reliability metrics, and capability consistency across multiple runs. Ideal for production monitoring and identifying subtle failure modes in advanced AI agents.

A comprehensive guide to setting up production-grade observability for LangChain applications. It integrates LangSmith tracing for zero-code debugging, custom callbacks for detailed metric collection (latency, token usage, error rate), and Prometheus exporters for robust alerting. This allows developers to monitor LLM pipelines, track performance metrics, and manage operational health in real-time.

LangSmith centralizes tracing, evaluation, and monitoring for LLM applications, letting teams debug prompts, run regression tests, collect latency/cost metrics, and integrate with OpenAI, Anthropic, LangChain, or LlamaIndex to keep production systems observable.

Master comprehensive strategies for evaluating Large Language Model (LLM) applications. This skill covers automated metrics (BLEU, ROUGE, BERTScore), structured human evaluation dimensions, and advanced LLM-as-Judge techniques. Use it to systematically measure model performance, compare prompts, detect regressions, and build confidence in production systems.

This skill thoroughly inspects and validates Vertex AI Agent Engine deployments. It assesses critical components such as the Code Execution Sandbox, Memory Bank, A2A protocol compliance, security posture, and performance metrics. The tool generates weighted production-readiness scores (0-100%) and provides detailed, actionable recommendations to ensure the deployment is robust, secure, and ready for production use.

Use the Recommendation Canvas to vet AI product concepts by mapping business and customer outcomes, hypotheses, positioning, risks, and success metrics so you can make defensible investment recommendations.

A comprehensive guide to building production-grade evaluation systems for Large Language Models (LLMs). This skill covers advanced methodologies, including Direct Scoring and Pairwise Comparison, alongside critical techniques for mitigating systematic biases (e.g., Position Bias, Length Bias). Learn how to select appropriate metrics and structure prompts to ensure reliable, consistent, and objective quality assessment across various AI applications.

A logging wrapper designed to instrument and monitor calls to the Claude API. It automatically tracks critical metrics including token usage, API latency, estimated cost, and error details. Essential for implementing robust observability patterns and setting up proactive alerts in production environments.