Generative AI

A practical guide to eliminating Ollama cold-start latency: how keep-alive works and why it matters, setting keep_alive per-request to -1 for permanent loading or 0 for immediate unloading, setting OLLAMA_KEEP_ALIVE globally, pre-loading models at application startup with a minimal dummy request, running multiple models simultaneously with OLLAMA_MAX_LOADED_MODELS, inspecting loaded models and VRAM usage via /api/ps, manually unloading models to free VRAM, and recommended settings for interactive chat, batch processing, multi-model RAG, and low-VRAM machines.

A practical guide to merging LoRA adapters into base models for production: merge_and_unload mechanics, merging multiple adapters sequentially, post-merge quantization to GGUF and AWQ, validating the merged model against the adapter-on-base setup, and when to keep adapters separate rather than merging.

A complete guide to Tabby, the self-hosted coding assistant built specifically for inline tab completions: how it differs from Continue and why dedicated completion models are faster and more accurate, Docker installation with NVIDIA GPU, choosing between StarCoder2 and DeepSeek-Coder models, VS Code, Neovim and JetBrains plugin setup, Docker Compose for persistent deployment, repository indexing for codebase-aware completions, monitoring acceptance rates in the built-in dashboard, and when to use Tabby versus Continue.

A practical guide to generating synthetic training data with LLMs: instruction-response pair generation, preference data for DPO, filtering and deduplication pipelines, optimal synthetic-to-real data mixing ratios, two-stage curriculum training, and the Evol-Instruct technique for scaling instruction complexity.

A complete reference for the Ollama REST API: listing and managing models with /api/tags, /api/pull, /api/delete, and /api/copy, running chat completions and raw text generation with /api/chat and /api/generate, generating embeddings with /api/embeddings, inspecting running models and VRAM usage with /api/ps, getting model details and Modelfiles with /api/show, creating custom models programmatically with /api/create, all key inference options, parsing the streaming response format including performance statistics, and health check patterns.

A practical guide to PyTorch CUDA memory management: how the caching allocator works, reading memory_stats and memory_summary, finding memory leaks with allocation snapshots, what empty_cache actually does, fixing fragmentation with expandable_segments, zero_grad set_to_none, and a systematic approach to diagnosing OOM errors at training time.

A practical guide to Mistral Nemo 12B for local use: what makes it distinctive including a native 128K context window, strong multilingual support, and the efficient Tekken tokeniser, hardware requirements at each quantisation level, running it with Ollama and configuring a 32K context Modelfile, benchmarking it against Llama 3.2 8B for your specific tasks, the four scenarios where its VRAM premium is justified, and how it compares to Mistral 7B and Mixtral 8x7B.

A practical guide to fine-tuning embedding models with contrastive learning: the Multiple Negatives Ranking Loss objective, building training datasets with synthetic query generation, hard negative mining, Matryoshka Representation Learning for flexible dimensions, evaluation with InformationRetrievalEvaluator, and when domain adaptation is actually worth the engineering cost.

A practical guide to connecting local LLMs to Obsidian for AI-powered note-taking: setting up Smart Connections with nomic-embed-text embeddings to search and chat across your entire vault, using the Ollama plugin for in-note summarisation and writing assistance, the BMO Chatbot for a persistent sidebar assistant, practical workflows for daily note processing, research synthesis, idea generation, and meeting note cleanup, and context window configuration for long notes.

A practical guide to using torch.profiler for GPU bottleneck analysis: profiler setup and schedule configuration, reading TensorBoard traces, identifying DataLoader gaps and CPU-GPU sync stalls, memory profiling, CUDA kernel occupancy, distributed training profiling, and a step-by-step workflow for translating profiler output into concrete training optimizations.