LLM inference latency comes from two distinct phases with different bottlenecks. A practical guide to KV cache management, continuous batching, quantization, speculative decoding, and prefix caching — and how to combine them for your specific latency vs throughput target.
Prompt engineering alone is not enough for reliable JSON output. A practical guide to constrained decoding with Outlines and vLLM guided decoding, Pydantic + instructor for OpenAI, semantic validation with retry logic, and when to fine-tune a smaller model for your schema.
A practical guide to writing custom CUDA kernels for PyTorch: project setup, the C++/CUDA extension mechanism, JIT compilation for fast iteration, autograd integration with gradcheck, thread indexing, memory coalescing, dtype dispatch, and debugging with compute-sanitizer.
A practical guide to Ollama Modelfiles: creating custom named models with persistent system prompts, setting temperature, context window, stop sequences and other inference parameters, four ready-to-use Modelfile templates for code review, JSON output, document summarisation, and low-RAM setups, using custom models through the Ollama REST API, seeding few-shot examples with MESSAGE, exporting and sharing Modelfiles with teammates, and the most common gotchas around context window memory, stop tokens, and reproducibility.
Most RAG failures are retrieval failures. A practical guide to the techniques that actually move the needle: hybrid BM25 + dense retrieval, reciprocal rank fusion, cross-encoder reranking, HyDE query transformation, and chunking strategies — with retrieval evaluation built in.
Tensor parallelism and pipeline parallelism split models across GPUs in fundamentally different ways with different latency and throughput implications. A practical guide covering how each works, NVLink vs InfiniBand trade-offs, and how they combine for 70B+ multi-node serving.
Off-the-shelf embedding models underperform on domain-specific retrieval. A practical guide to synthetic query generation, hard negative mining, contrastive fine-tuning, and Matryoshka training — with realistic expectations on NDCG@10 improvement.
AWQ, GPTQ, and bitsandbytes each quantize LLM weights differently. A practical comparison covering how each works, quality at 4-bit and 8-bit, inference speed, and when to use which — including the right choice for QLoRA training vs production serving.
FlashAttention-2 delivers 2–4x faster attention and eliminates O(N²) memory with a single argument change. A practical guide to enabling it in HuggingFace, PyTorch SDPA, and fine-tuning pipelines — including attention mask compatibility and where it helps most.
RLHF with PPO is powerful but complex. DPO achieves comparable alignment with a fraction of the engineering overhead. A practical comparison of PPO, DPO, and newer variants like SimPO and ORPO — covering when to use each and how to build a preference dataset that actually works.
