Skill UI

This is a high-level Software Development Kit (SDK) designed for managing the full lifecycle of AI projects within Azure AI Foundry. Built for Java, it provides robust client interfaces for core AI operations, including managing connections to Azure resources, uploading and managing datasets, creating and updating search indexes, running model evaluations, and handling deployments. It streamlines complex AI workflow orchestration.

This is a high-level Software Development Kit (SDK) designed for managing the full lifecycle of AI projects within Azure AI Foundry. Built for Java, it provides robust client interfaces for core AI operations, including managing connections to Azure resources, uploading and managing datasets, creating and updating search indexes, running model evaluations, and handling deployments. It streamlines complex AI workflow orchestration.

This is the .NET SDK for managing Arize AI Observability and Evaluation resources within Azure Resource Manager (ARM). It provides programmatic access to create, read, update, and delete AI organization resources. It is essential for MLOps professionals and developers who need to monitor, track, and evaluate the performance of AI models deployed on Azure.

Formal evaluation framework for Claude Code sessions that applies evaluation-driven development to define capability and regression checks, run deterministic and model graders, and track pass@k/pass^k reliability metrics before shipping.

A comprehensive toolkit for evaluating the performance of code generation models. It supports benchmarking against industry standards like HumanEval, MBPP, and MultiPL-E across multiple languages. Use this tool to quantitatively compare different LLM coding abilities and measure code synthesis quality using pass@k metrics.

Evaluates large language models across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) with standardized prompts and metrics, making it easy to benchmark HuggingFace or API models, compare releases, and track training progress.

Master comprehensive strategies for evaluating Large Language Model (LLM) applications. This skill covers automated metrics (BLEU, ROUGE, BERTScore), structured human evaluation dimensions, and advanced LLM-as-Judge techniques. Use it to systematically measure model performance, compare prompts, detect regressions, and build confidence in production systems.

Automates guidance for evaluating machine learning models, covering best practices, sample code, and validation patterns triggered by mentions of model evaluation metrics.

This skill specializes in designing, optimizing, and evaluating prompts for Large Language Models (LLMs). It allows users to generate optimized prompt templates, structured output schemas (e.g., JSON mode, function calling), and comprehensive test suites. Use this skill when building new LLM applications, implementing complex reasoning patterns like Chain-of-Thought, defining system guardrails, or building robust evaluation frameworks to measure and improve model performance.

This comprehensive workflow guides the end-to-end implementation of Retrieval-Augmented Generation (RAG) systems. It covers critical stages, including selecting optimal embedding models, setting up vector databases, designing chunking strategies, implementing hybrid retrieval, integrating Large Language Models (LLMs), and performing rigorous evaluation. Use this when building knowledge-grounded AI applications or advanced semantic search systems.

A comprehensive, model-agnostic toolkit for advanced LLM development. It enables iterative prompt optimization through static analysis (token usage, clarity), measures critical RAG metrics (relevance, faithfulness, coverage), and validates complex agent workflows and configurations. Essential for rigorous LLM quality assurance and cost management.

Scaffolds production-ready generative AI agents on Vertex AI with Gemini models, multi-modal tools, RAG, and Cloud Run/Endpoint deployment, including observability and evaluation for enterprise scenarios.