Three Approaches, One Question
When an LLM does not behave the way you need it to, three distinct strategies are available to improve its performance: prompting (changing what you say to the model at inference time), retrieval-augmented generation (providing relevant external information at inference time), and fine-tuning (changing the model’s weights through training). These are not competing alternatives — they address different problems, operate at different costs and timescales, and are often combined in production systems. Understanding precisely what each one does and does not solve is the foundation of intelligent LLM application design.
The most common mistake is treating fine-tuning as the solution to problems that prompting or RAG would solve more effectively, faster, and at lower cost. The second most common mistake is attempting to use RAG for knowledge that should be encoded in the model’s behaviour through prompting or fine-tuning. Getting this decision right at the start of a project saves weeks of work and produces better outcomes.
Prompting: What It Solves
Prompting adjusts model behaviour without changing anything persistent — every change lives in the input and costs tokens on every request. Prompting is the right solution for: behavioural instructions (tone, format, scope, persona), context provision (background information the model needs for this specific task), task specification (exactly what you want the model to do and how), output structure (how the response should be formatted), and constraint enforcement (what the model should and should not do). These are all things the model can do based on its existing capabilities — prompting guides how those capabilities are applied.
Prompting is not the right solution for: injecting up-to-date information the model was not trained on, providing access to private or proprietary information that should not be sent on every request, or changing the model’s deep behavioural defaults in ways that persist without the prompt being present. If removing the instruction from the prompt makes the model revert to undesired behaviour, you have a prompting dependency that may be better addressed through fine-tuning. If the model’s output quality depends on information that is not always available in the prompt, you have an information access problem that RAG solves.
RAG: What It Solves
Retrieval-augmented generation retrieves relevant information from an external corpus and provides it to the model alongside the user’s query. RAG solves the information access problem: when the model needs specific, current, or proprietary information to answer a question correctly, RAG provides it rather than relying on training knowledge that may be outdated, incomplete, or simply not present. RAG is the right solution for: answering questions about documents the model was not trained on, providing access to information that updates frequently, grounding responses in specific sources with citations, enabling responses about proprietary or confidential information without that information being in training data, and scaling to large knowledge bases that would not fit in a single context window.
RAG is not the right solution for: changing how the model behaves or responds (prompting does this), teaching the model new skills or output formats (fine-tuning does this), or providing information that the model already knows from training (unnecessary overhead). The diagnostic question is: does the model fail because it lacks information, or because it has information but applies it incorrectly? Information access failures call for RAG; behavioural failures call for prompting or fine-tuning.
Fine-Tuning: What It Solves
Fine-tuning updates the model’s weights through training on examples, permanently changing its default behaviour for that task. Fine-tuning is the right solution for: teaching highly specific output formats that prompting cannot reliably enforce, encoding domain expertise that is difficult to express as retrievable documents, adapting model tone and style at a fundamental level that persists without explicit prompting, making a smaller model perform like a larger one on a specific narrow task, and high-volume applications where the token cost of extensive prompting is significant. Fine-tuning requires high-quality training data, a training run, and an evaluation pipeline — it is a larger investment than prompting or RAG setup, and should be the last resort rather than the first instinct.
Figure 1 — When to Use Each Approach
The Decision Sequence
The correct order of operations is always the same: start with prompting, add RAG if needed, fine-tune only if both are insufficient. This sequence reflects both the cost structure (prompting is cheapest to iterate, fine-tuning is most expensive) and the failure mode hierarchy (most problems are prompting problems; fewer are information access problems; fewer still require weight updates). Working through the sequence also produces the training data needed for fine-tuning as a by-product: the examples that prompting and RAG fail to handle correctly, documented with their correct outputs, become the fine-tuning dataset.
Diagnosing which step is the bottleneck requires honest evaluation. Take 20–30 representative examples of failures and categorise them: does the model have the information needed to answer correctly? If no, the problem is information access — try RAG. If yes, does the model apply that information correctly when prompted well? If no, the problem is behavioural — improve the prompt. If yes but inconsistently, the problem may be format or style — try fine-tuning. This diagnostic process takes an hour and reliably identifies where engineering investment should go before any actual implementation work begins.
Combining All Three
Production LLM applications often use all three approaches together in layers. Prompting establishes the model’s persona, constraints, and output format. RAG provides the relevant document context for each query. Fine-tuning has adapted the base model to produce outputs in the organisation’s specific style and format with greater reliability than prompting alone achieves. Each layer addresses a distinct aspect of the quality challenge that the others cannot solve alone. The combination delivers quality that no single approach matches — consistent behaviour from fine-tuning, current information from RAG, and precise task instructions from prompting — at a total cost that is still lower than using a frontier model with extensive prompting for everything.
Common Mistakes in This Decision
Reaching for fine-tuning first is the most expensive mistake. A team that spends three weeks collecting data, training, and evaluating a fine-tuned model to solve a problem that better prompting would have resolved in a day has wasted significant resources. Treating RAG as a replacement for fine-tuning is equally problematic in the other direction: RAG cannot teach the model new skills or change its output style — it only provides information. And conflating these approaches — trying to use prompting to ground responses in external documents that are not in the prompt, or using RAG to fix behavioural problems — creates systems that are expensive to operate and still fail on the actual problem. Clarity about what each approach does and does not solve prevents these expensive mismatches.
Cost and Time Comparison
The investment required for each approach differs dramatically. Prompting can be done in minutes to hours — write a better prompt, test it, deploy it. The only cost is engineering time and the token cost of any additional prompt content. RAG setup takes days to weeks — build the ingestion pipeline, embed the corpus, set up retrieval, tune retrieval quality. Ongoing costs include embedding new content, vector database hosting, and the retrieval overhead added to each query. Fine-tuning takes weeks to months — collect and curate training data, run training, evaluate the fine-tuned model, iterate. The upfront cost is high but inference costs may be lower if fine-tuning allows use of a smaller model. These cost differences make the sequence of prompting-first, RAG-second, fine-tuning-last not just intellectually correct but economically rational. The cheapest solution that meets your quality threshold is always the right one.
When the Answer Changes Over Time
The right balance between prompting, RAG, and fine-tuning often evolves as an application matures. Many applications start with prompting-only because it is the fastest path to a working prototype. As the corpus of relevant documents grows, RAG is added. As volume increases and format consistency becomes important, fine-tuning of a smaller model is evaluated. This progression is not a sign of early poor design — it reflects appropriately sequencing the investments to match the actual quality constraints encountered in production rather than speculating about them upfront. The application that starts with fine-tuning before validating the core value proposition has made a large investment before understanding whether the product concept is correct. The application that starts with prompting, validates the concept, and adds RAG and fine-tuning incrementally as specific quality gaps are identified has spent the same total investment more efficiently and with less risk.
The teams that understand this sequencing — and resist the temptation to over-engineer early — consistently ship better applications faster and with lower total investment than those that start with the most complex solution. Prompting remains your most powerful lever. RAG extends your reach to knowledge the model does not have. Fine-tuning refines the model’s core behaviour when both of those are insufficient. Used in sequence, they cover every dimension of LLM quality improvement available to application developers today.