Peter Song 
Real-time machine learning has transformed from a luxury to a necessity for modern applications. Whether powering fraud detection systems that must respond within milliseconds, recommendation engines that adapt to user behavior instantly, or dynamic pricing algorithms that adjust to market conditions in real-time, the ability to compute and serve fresh features is critical. However, bridging … Read more
The terminology surrounding modern AI can be bewildering, with terms like “large language models,” “transformers,” “GPT,” and “neural networks” often used interchangeably or inconsistently across different contexts. Among the most common sources of confusion is the relationship between “large language models” (LLMs) and “transformers”—are they the same thing? Different things? Is one a subset of … Read more
The rise of large language models has introduced practitioners to two fundamentally different processes for creating AI systems: training from scratch and fine-tuning pre-trained models. While both involve adjusting model parameters through gradient descent, the scale, purpose, cost, and outcomes differ so dramatically that they represent entirely different approaches to model development. Training builds a … Read more
The lifecycle of a large language model splits into two fundamentally distinct phases: training and inference. While both involve passing data through neural networks, the computational demands, objectives, constraints, and optimization strategies differ so dramatically that they might as well be separate disciplines. Training is the expensive, time-intensive process of teaching a model to understand … Read more
Large language models like GPT-4, Claude, and Llama generate text through a process that appears seamless to users but actually unfolds in two distinct computational phases: the prefill phase and the decode phase. Understanding these two steps is fundamental to grasping how LLMs work, why they behave the way they do, and what engineering challenges … Read more
Large language models have achieved remarkable capabilities, but their computational demands create a fundamental tension between performance and accessibility. A 70-billion parameter model in standard FP16 precision requires approximately 140GB of memory—far exceeding what’s available on consumer GPUs and even challenging high-end datacenter hardware. Quantization techniques address this challenge by reducing the numerical precision of … Read more
Nearest neighbors search stands as one of the most fundamental operations in machine learning and data science, underpinning everything from recommendation systems to anomaly detection, from image retrieval to dimensionality reduction techniques like t-SNE. Yet the seemingly simple task of finding the k closest points to a query point becomes computationally challenging as datasets grow … Read more
Reinforcement Learning from Human Feedback (RLHF) has emerged as the critical technique behind the most capable language models in production today. While the conceptual framework appears straightforward—collect human preferences, train a reward model, optimize the policy—building RLHF pipelines that scale to enterprise demands requires navigating a complex landscape of infrastructure challenges, data quality concerns, and … Read more
In the landscape of machine learning, one of the most fundamental yet often misunderstood distinctions lies between probabilistic and deterministic algorithms. This divide isn’t merely a technical curiosity—it shapes how models make predictions, quantify uncertainty, handle ambiguous data, and ultimately serve real-world applications. Understanding when to employ each approach can be the difference between a … Read more
Vector similarity metrics form the backbone of modern machine learning systems, from recommendation engines that suggest your next favorite movie to search engines that retrieve relevant documents from billions of candidates. Yet the choice between cosine similarity, dot product, and Euclidean distance profoundly affects results in ways that aren’t immediately obvious. A recommendation system using … Read more
