Skill UI

This is a high-level Software Development Kit (SDK) designed for managing the full lifecycle of AI projects within Azure AI Foundry. Built for Java, it provides robust client interfaces for core AI operations, including managing connections to Azure resources, uploading and managing datasets, creating and updating search indexes, running model evaluations, and handling deployments. It streamlines complex AI workflow orchestration.

This is a high-level Software Development Kit (SDK) designed for managing the full lifecycle of AI projects within Azure AI Foundry. Built for Java, it provides robust client interfaces for core AI operations, including managing connections to Azure resources, uploading and managing datasets, creating and updating search indexes, running model evaluations, and handling deployments. It streamlines complex AI workflow orchestration.

This is the .NET SDK for managing Arize AI Observability and Evaluation resources within Azure Resource Manager (ARM). It provides programmatic access to create, read, update, and delete AI organization resources. It is essential for MLOps professionals and developers who need to monitor, track, and evaluate the performance of AI models deployed on Azure.

Formal evaluation framework for Claude Code sessions that applies evaluation-driven development to define capability and regression checks, run deterministic and model graders, and track pass@k/pass^k reliability metrics before shipping.

A comprehensive toolkit for evaluating the performance of code generation models. It supports benchmarking against industry standards like HumanEval, MBPP, and MultiPL-E across multiple languages. Use this tool to quantitatively compare different LLM coding abilities and measure code synthesis quality using pass@k metrics.

Evaluates large language models across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) with standardized prompts and metrics, making it easy to benchmark HuggingFace or API models, compare releases, and track training progress.

Master comprehensive strategies for evaluating Large Language Model (LLM) applications. This skill covers automated metrics (BLEU, ROUGE, BERTScore), structured human evaluation dimensions, and advanced LLM-as-Judge techniques. Use it to systematically measure model performance, compare prompts, detect regressions, and build confidence in production systems.

Automates guidance for evaluating machine learning models, covering best practices, sample code, and validation patterns triggered by mentions of model evaluation metrics.

This skill specializes in designing, optimizing, and evaluating prompts for Large Language Models (LLMs). It allows users to generate optimized prompt templates, structured output schemas (e.g., JSON mode, function calling), and comprehensive test suites. Use this skill when building new LLM applications, implementing complex reasoning patterns like Chain-of-Thought, defining system guardrails, or building robust evaluation frameworks to measure and improve model performance.

This comprehensive workflow guides the end-to-end implementation of Retrieval-Augmented Generation (RAG) systems. It covers critical stages, including selecting optimal embedding models, setting up vector databases, designing chunking strategies, implementing hybrid retrieval, integrating Large Language Models (LLMs), and performing rigorous evaluation. Use this when building knowledge-grounded AI applications or advanced semantic search systems.

Scaffolds production-ready generative AI agents on Vertex AI with Gemini models, multi-modal tools, RAG, and Cloud Run/Endpoint deployment, including observability and evaluation for enterprise scenarios.

A comprehensive guide to building production-grade evaluation systems for Large Language Models (LLMs). This skill covers advanced methodologies, including Direct Scoring and Pairwise Comparison, alongside critical techniques for mitigating systematic biases (e.g., Position Bias, Length Bias). Learn how to select appropriate metrics and structure prompts to ensure reliable, consistent, and objective quality assessment across various AI applications.