Skill UI

Activation-aware weight quantization that compresses 7B-70B models to 4-bit with minimal accuracy loss, delivering ~3x inference speedups and memory reductions for constrained GPU deployments while integrating with vLLM and Marlin kernels.

Guides ML engineers through dataset prep, PEFT strategy selection, LoRA/QLoRA adapter setup, training monitoring, and deployment checks for practical LLM fine-tuning, quantization, and adapter merge workflows.

This guide details the use of the GGUF format and quantization techniques for running large language models (LLMs) efficiently. GGUF enables optimal inference on consumer hardware, Apple Silicon, and systems requiring CPU-only operation. By compressing models using various K-quant methods (like Q4_K_M), developers can significantly reduce memory footprint and hardware requirements while maintaining high performance.

GPTQ enables post-training 4-bit quantization for large LLMs, delivering up to 4× memory reduction and 3-4× faster inference with under 2% perplexity loss on consumer GPUs through AutoGPTQ, PEFT, and Transformer integrations.

Provides calibration-free weight quantization for LLMs down to 1-bit precision with multiple optimized backends, ideal for fast deployments with vLLM/HuggingFace and LoRA/PEFT fine-tuning without collecting calibration data.

llama.cpp is a pure C/C++ library designed for efficient LLM inference, optimized to run on resource-constrained environments. It excels in scenarios where high-end NVIDIA GPUs (CUDA) are unavailable, making it ideal for Apple Silicon Macs, AMD/Intel GPUs, or edge deployment devices like Raspberry Pi. It supports GGUF quantization formats, enabling significant memory reduction and speed improvements for local, cross-platform AI applications.

Automates guidance for deploying machine learning models with quantization, covering tooling, best practices, and production readiness for inference, MLops, and monitoring tasks.

Quantizes HuggingFace LLMs to 8-bit or 4-bit with bitsandbytes, cutting memory by up to 75% while keeping accuracy, and supports QLoRA fine-tuning plus 8-bit optimizers for faster, memory-efficient training.

An expert guide for deploying and optimizing Large Language Models (LLMs) entirely on local hardware. Mastering the entire local AI ecosystem, including inference engines like Ollama, vLLM, and llama.cpp. Provides deep knowledge on quantization formats (GGUF, EXL2, AWQ), VRAM calculation, and ensuring privacy-first, offline deployment for state-of-the-art models.

A comprehensive guide for optimizing cloud GPU expenditures on CoreWeave. Learn strategies like right-sizing instances, implementing scale-to-zero for development environments, and using quantization techniques (e.g., AWQ, GPTQ) to fit large models onto smaller, cost-effective GPUs, ensuring maximum efficiency and budget adherence for AI/ML workloads.

A comprehensive guide for diagnosing and resolving slow search performance in Qdrant. This resource covers issues related to low QPS (throughput), high latency for single queries, and slow filtered searches. Solutions include tuning HNSW parameters, managing memory pressure, implementing quantization, optimizing indexing, and configuring cluster settings for maximum efficiency.

A comprehensive guide and skill for deploying large language models (LLMs) locally using the llama.cpp framework and GGUF quantization format. It covers searching the Hugging Face Hub, selecting optimal quantizations (e.g., Q4_K_M), running local servers, and achieving OpenAI-compatible API endpoints. Supports various hardware backends including CPU, Mac Metal, CUDA, and ROCm. Ideal for developers needing offline, private LLM inference.