Skill UI

Activation-aware weight quantization that compresses 7B-70B models to 4-bit with minimal accuracy loss, delivering ~3x inference speedups and memory reductions for constrained GPU deployments while integrating with vLLM and Marlin kernels.

A comprehensive guide to understanding the complete state (context) available to Large Language Models during inference. Covers core components like System Prompts, Tool Definitions, Message History, and Retrieval Augmented Generation (RAG). Learning to engineer context is crucial for building robust, efficient, and reliable AI agent systems by managing the attention budget and ensuring high-signal information delivery.

This guide details the use of the GGUF format and quantization techniques for running large language models (LLMs) efficiently. GGUF enables optimal inference on consumer hardware, Apple Silicon, and systems requiring CPU-only operation. By compressing models using various K-quant methods (like Q4_K_M), developers can significantly reduce memory footprint and hardware requirements while maintaining high performance.

GPTQ enables post-training 4-bit quantization for large LLMs, delivering up to 4× memory reduction and 3-4× faster inference with under 2% perplexity loss on consumer GPUs through AutoGPTQ, PEFT, and Transformer integrations.

This guide provides comprehensive instructions for installing and configuring the official Groq SDK in both Python and TypeScript environments. It covers dependency setup, secure API key management using environment variables, and provides code examples to verify a successful connection to Groq's ultra-fast LLM inference API. Essential for initiating any project powered by Groq.

This guide details a production-ready reference architecture for applications built on Groq's LPU inference API. It covers sophisticated model routing based on latency, quality, or cost needs, implementing streaming pipelines, and establishing robust multi-provider fallback chains (e.g., Groq primary to OpenAI backup). Ideal for designing and reviewing complex, resilient LLM systems.

Knowledge Distillation is a crucial technique used to compress large, high-performing Language Models (Teacher) into smaller, more efficient Student models. This allows developers to deploy powerful AI capabilities (like those of GPT-4) on resource-constrained devices while significantly reducing inference costs. The method uses advanced techniques such as soft targets, temperature scaling, and Reverse KL Divergence (MiniLLM) to transfer knowledge effectively.

llama.cpp is a pure C/C++ library designed for efficient LLM inference, optimized to run on resource-constrained environments. It excels in scenarios where high-end NVIDIA GPUs (CUDA) are unavailable, making it ideal for Apple Silicon Macs, AMD/Intel GPUs, or edge deployment devices like Raspberry Pi. It supports GGUF quantization formats, enabling significant memory reduction and speed improvements for local, cross-platform AI applications.

LlamaGuard is Meta’s 7–8B safety-specialized LLM that filters both prompts and responses by classifying six threat categories, enabling fast inference via vLLM/SageMaker and integration into NeMo Guardrails for end-to-end moderation.

Compress large language models quickly using Wanda, SparseGPT, and N:M pruning to enforce hardware-friendly sparsity, enabling faster inference and deployment without retraining.

This skill automates the setup and configuration of Ollama, allowing users to deploy Large Language Models (LLMs) locally. It performs system assessment (OS, GPU, RAM), selects optimal models based on hardware availability, and handles installation across macOS, Linux, and Docker. Use this when you require free, self-hosted AI alternatives, need offline inference capabilities, or wish to eliminate reliance on costly hosted API services.

Outlines guarantees valid JSON/XML/code outputs by applying CFG/FSM-constrained token filtering, Pydantic schemas, and grammar-level sampling while supporting local Transformers, llama.cpp, and vLLM backends for fast inference.