The Fine-Tuning Temptation
Fine-tuning is the most over-prescribed solution in LLM application development. When a model produces outputs that do not match expectations, the intuitive response is to train it on better examples. But in the majority of cases, the problem is not that the model lacks the capability — it is that the instructions it was given were insufficient to elicit the capability it already has. Prompt engineering fixes this at a fraction of the cost and time of fine-tuning, and produces results immediately rather than after a training and evaluation cycle.
This guide explains precisely when fine-tuning is the right solution, when better prompting is the right solution, and how to tell the difference before investing in a training run. It also covers the practical mechanics of when fine-tuning genuinely adds value that prompting cannot — because those cases do exist and are important to identify correctly.
What Fine-Tuning Actually Does
Fine-tuning updates a model’s weights by training on a dataset of examples, adjusting the model’s internal representations to produce outputs more consistent with that training data. It teaches the model new patterns — styles, formats, domain-specific knowledge, or specific response behaviours — that become part of the model’s default behaviour rather than requiring explicit instruction in every prompt.
There are two main fine-tuning approaches. Full fine-tuning updates all of the model’s weights and requires the most compute and data. Parameter-efficient methods like LoRA (Low-Rank Adaptation) and QLoRA update a small fraction of the weights — often less than 1% — while producing most of the behavioural adaptation of full fine-tuning. LoRA at rank 8–16 on a 7B or 13B model fits on a single consumer GPU and is the practical standard for most fine-tuning projects. The resulting fine-tuned model is your base model with the LoRA adapter applied — you can switch adapters to load different fine-tuned specialisations of the same base model.
When Prompting Solves the Problem
Prompting is the right solution in the majority of cases where outputs do not meet expectations. Several specific problems that appear to require fine-tuning are actually prompting problems. The model ignores instructions: Often caused by instructions that are vague, contradictory, or buried in a long prompt. Rewrite with clearer, more specific instructions at the beginning of the system prompt. The model produces the wrong format: Explicit format examples, tool forcing for JSON output, and clear output templates solve most format problems without training. The model answers incorrectly on domain-specific questions: If the information exists in documents you can provide, RAG is almost always better than fine-tuning — it does not require training, the information can be updated without retraining, and it provides citations that fine-tuning cannot. The model tone is wrong: Tone and style are highly controllable through prompting with examples. If you have 5–10 examples of the tone you want, include them in the prompt as few-shot examples. The model is too verbose or too brief: Explicit length instructions solve this reliably. If your prompt says “answer in 2–3 sentences,” the model will do so.
Before pursuing fine-tuning, invest seriously in prompt engineering first. A rigorous prompt engineering process — testing variations, adding examples, clarifying instructions, testing on edge cases — resolves the majority of LLM quality problems. Fine-tuning a poorly-prompted model produces a fine-tuned poorly-prompted model. The prompt engineering investment is not wasted even if you eventually fine-tune — it produces the training examples and quality criteria you need for the fine-tuning process.
When Fine-Tuning Is the Right Answer
The task requires consistent output in a highly specific format that prompting cannot reliably produce. Some output formats are sufficiently unusual or precise that even detailed prompting produces occasional format violations. Fine-tuning on 500–1000 examples of correct format reliably teaches the model the format constraints, reducing format error rates from 5–10% (with prompting) to under 1%. This matters when downstream systems consume the output programmatically and format errors cause failures.
You need to inject substantial proprietary knowledge that cannot be retrieved at inference time. RAG is usually better for retrievable information. But some knowledge is better baked into the model’s weights — complex domain-specific reasoning patterns, procedural knowledge about how to approach a class of problem, or tacit knowledge that is hard to express as retrievable documents. A medical coding model that needs to apply complex ICD-10 coding logic may benefit from fine-tuning on thousands of correctly-coded examples rather than retrieving coding guidelines at inference time.
Latency or cost requires a smaller model to match the quality of a larger one. Fine-tuning a small model (7B–13B) on domain-specific examples can close much of the quality gap between it and a larger general-purpose model on that specific domain, at a fraction of the inference cost and latency. If your use case is well-defined and high-volume, fine-tuning a smaller model may produce the best quality-cost-latency combination.
You have thousands of high-quality labelled examples and a clearly defined, narrow task. Fine-tuning requires good data. If you have fewer than 100 examples, fine-tuning is almost certainly the wrong approach — the model will not learn generalised patterns from that volume. If you have 500–2000 high-quality examples of a specific, well-defined task, fine-tuning begins to be viable. If you have 10,000+ examples, fine-tuning is likely to produce meaningful gains over prompting for that specific task.
Figure 1 — Prompt Engineering vs. Fine-Tuning Decision Guide
The Data Requirements for Successful Fine-Tuning
Data quality matters far more than data quantity in fine-tuning. One hundred carefully curated, high-quality examples produce better results than one thousand mediocre ones. Quality means: the input-output pairs represent exactly the behaviour you want to train, without edge cases, inconsistencies, or outputs you would not want the model to reproduce. Every example in your training set teaches the model something. Bad examples teach the model bad behaviours. Before training on any dataset, review a random sample of 50–100 examples manually. If you find yourself uncomfortable with any of the outputs in that sample, the dataset is not ready for training.
For instruction-following and style fine-tuning, 200–500 high-quality examples are usually sufficient for meaningful improvement. For complex domain-specific reasoning, 1,000–5,000 examples are typically needed. For tasks requiring broad domain coverage, 10,000+ examples may be necessary. If you do not have enough quality data to fine-tune, the right answer is not to fine-tune on insufficient data — it is to collect more data through human annotation, or to continue using prompting until the data volume is sufficient.
Evaluating Whether Fine-Tuning Worked
Fine-tuning must be evaluated on a held-out test set that was not used during training. Use the same evaluation methodology you used to establish your pre-fine-tuning baseline. Measure quality improvement on the target task, quality degradation on general capabilities (fine-tuning can cause catastrophic forgetting on capabilities outside the training distribution), format consistency, and edge case behaviour. A fine-tuned model that improves target task performance by 15% while degrading general capability by 30% may be a net negative depending on how often you need the general capability. Compare the fine-tuned model against a well-prompted version of the same base model — not just against the baseline unprompted model — to isolate the incremental value of fine-tuning beyond what prompting achieves.
The Verdict: A Simple Decision Rule
Try prompting first — always. If you have done serious prompt engineering (multiple iterations, few-shot examples, structured instructions) and the output quality still falls below your threshold, then ask: do I have 500+ high-quality training examples? Is the task narrow and well-defined? Is the failure mode one that fine-tuning addresses (style, format, domain knowledge) rather than one it doesn’t (hallucination, reasoning failures, information retrieval)? If yes to all three, fine-tuning is worth evaluating. If no to any of them, continue improving your prompts or consider RAG before investing in a fine-tuning project. The best fine-tuning projects start with excellent prompting — the prompt engineering work defines what good outputs look like, which is exactly what you need to generate quality training data for the fine-tuning run.
Continued Learning and Model Maintenance
Fine-tuning is not a one-time event. If your fine-tuned model is in production, you need a plan for maintaining it over time. The base model it was trained on will be updated or deprecated, requiring you to either re-fine-tune on the new base or migrate to a different approach. Your production data distribution will shift, and examples that were well-represented in your training set may become less common while new patterns emerge. Your quality requirements may evolve. Build re-training triggers into your fine-tuning governance: automatic re-evaluation when the base model is updated, quarterly review of production quality metrics to catch distribution shift, and a data flywheel that captures low-quality production outputs for human review and potential inclusion in the next training run. Fine-tuned models that are maintained this way continuously improve; those that are trained once and left static gradually become less appropriate as the world and your application evolve. The investment in continuous improvement infrastructure is what turns a one-time fine-tuning project into a durable capability that compounds in value over time.