Why Alignment Matters for Builders
AI alignment is the challenge of ensuring that AI systems do what their designers and users intend, and nothing they do not. For LLM application builders, alignment is not an abstract research concern — it is a practical engineering reality that affects every system you deploy. The models you are building on have been trained with specific alignment techniques that shape their behaviour, and understanding those techniques helps you predict what your model will and will not do, debug unexpected behaviours, and make better decisions about safety controls for your application.
This guide covers the main alignment techniques used in production LLMs — RLHF, Constitutional AI, and direct preference optimisation — what they achieve, where they fall short, and what they mean for builders deploying these models in real applications.
The Alignment Problem in Plain Terms
A raw language model trained only on next-token prediction will produce whatever text is statistically likely given its training data — which includes harmful, misleading, and socially unacceptable content, because that content exists in the training data. Alignment techniques shape the model’s behaviour beyond statistical text prediction toward outputs that are helpful, accurate, and safe. Without alignment, a capable language model is not a useful product — it is a text completion engine that will enthusiastically assist with any request, harmful or not.
The challenge is that alignment is hard to specify precisely. “Be helpful but not harmful” sounds simple but involves millions of judgment calls about what counts as harmful, how to balance helpfulness against harm in edge cases, and how to handle the enormous variety of human requests, contexts, and intentions. The major alignment approaches each represent a different answer to the question of how to make these specifications computationally tractable.
Reinforcement Learning from Human Feedback (RLHF)
RLHF is the most widely used alignment technique and the foundation of the alignment training in GPT models, Claude’s early versions, and most commercial LLMs. The process has three stages. First, supervised fine-tuning (SFT): the base model is fine-tuned on a dataset of human-written demonstrations of desirable outputs — examples of how a helpful, accurate, appropriate assistant should respond to various inputs. Second, reward model training: human raters compare pairs of model outputs and indicate which is better. A separate reward model is trained on these preferences to predict human preference ratings for any given output. Third, reinforcement learning: the language model is trained using the reward model’s predictions as a reward signal, using Proximal Policy Optimisation (PPO) to nudge the model toward outputs the reward model rates highly.
RLHF’s strengths are substantial: it captures nuanced human preferences that are difficult to specify as explicit rules, it scales with the amount of human feedback provided, and it has produced demonstrably safer and more helpful models than pure supervised fine-tuning. Its weaknesses are equally important: it is expensive (human raters are needed at scale), it is subject to reward hacking (the model learns to satisfy the reward model without necessarily producing genuinely good outputs), and human raters have biases that propagate into the model’s trained preferences.
Constitutional AI: Anthropic’s Approach
Constitutional AI (CAI), developed by Anthropic, addresses some of RLHF’s limitations by reducing dependence on human feedback for the harmlessness dimension of alignment. Instead of using human raters to label harmful outputs, CAI uses a set of explicit principles — the “constitution” — and asks the model itself to evaluate its outputs against those principles and revise them. The process has two phases. In the supervised learning phase, the model generates responses, critiques them against the constitutional principles (“does this response encourage harmful behaviour?”), revises them, and the revised responses become fine-tuning data. In the reinforcement learning phase, a reward model is trained using AI-generated preference labels based on constitutional principles rather than human ratings — called Reinforcement Learning from AI Feedback (RLAIF).
CAI’s advantages are transparency (the principles are explicit and auditable), scalability (AI feedback is cheaper than human feedback at scale), and consistency (AI raters apply the principles more consistently than human raters with varying personal standards). The limitation is that the constitution’s principles must be specified by humans, which means the alignment is only as good as the principles — and defining comprehensive, unambiguous principles that cover every edge case is genuinely difficult. For builders, CAI is relevant because it explains why Claude models tend to be explicit about their values and reasoning in a way that RLHF-trained models often are not: transparency about principles is built into the training process.
Figure 1 — The Three Layers of LLM Alignment
Direct Preference Optimisation (DPO)
Direct Preference Optimisation is a newer alignment technique that achieves results comparable to RLHF without training a separate reward model, making it significantly simpler and more stable to implement. DPO directly optimises the language model on human preference data — pairs of outputs where one is preferred over the other — using a mathematical formulation that shows the optimal policy can be expressed in closed form without a reinforcement learning loop. In practice, DPO training is more stable than PPO-based RLHF, requires less compute, and produces models with comparable or better alignment properties on many tasks. Many of the open-source model families released in 2024 and 2025, including Llama 3 instruction models, use DPO or variants (IPO, KTO) rather than full RLHF.
What Alignment Does and Does Not Guarantee
Understanding alignment’s limits is as important as understanding its capabilities. Alignment training makes models significantly less likely to produce harmful outputs — but it does not make this impossible. Adversarial inputs (jailbreaks), carefully constructed prompt sequences, or unusual phrasings can sometimes elicit outputs that the alignment training was supposed to prevent. This is an active research area and models improve with each generation, but no current alignment approach provides absolute guarantees. For builders, this means that alignment is a layer of defence, not a complete solution — applications with strict content requirements still need additional filtering, output monitoring, and human review of edge cases.
Alignment also affects helpfulness in ways that matter for application design. Aligned models are calibrated to refuse certain types of requests, add disclaimers to sensitive topics, and express uncertainty when appropriate — behaviours that are appropriate defaults for consumer products but may be over-cautious for some enterprise applications. The system prompt is the primary tool for adjusting this balance: you can give the model context about your application and users that allows it to be more or less cautious in specific ways, within the limits set by the model provider’s policies.
Practical Implications for Builders
Several alignment properties directly affect how you design and operate LLM applications. Instruction following vs. safety refusals: Aligned models balance following your instructions with their safety training. Very long system prompts with many instructions sometimes cause the model to prioritise safety training over specific instructions — keep system prompts focused on what you actually need. Sycophancy: RLHF-trained models learn that human raters tend to prefer agreeable responses, which can cause them to agree with incorrect statements in user messages rather than correcting them. For applications requiring accurate information, include explicit instructions to correct errors and disagree when necessary. Verbosity: Aligned models often produce longer responses than necessary because verbose, comprehensive responses tend to be rated higher by human evaluators. Explicit length instructions counteract this. Consistency: Alignment training is applied probabilistically — the same input may produce slightly different outputs across runs. For applications requiring consistent outputs, use temperature 0 and test for consistency explicitly.
The Future of Alignment
Alignment research is advancing rapidly. Scalable oversight — methods for supervising AI systems on tasks where human evaluators cannot easily verify correctness — is a major research direction as models become capable of tasks beyond easy human evaluation. Debate and amplification approaches use AI systems to help humans evaluate AI outputs more accurately. Interpretability research aims to understand what concepts and representations are encoded in model weights, enabling more targeted and verifiable alignment. Constitutional approaches are being extended with more sophisticated principle hierarchies. For builders, these research directions will gradually produce models that are better aligned, more consistent, and better able to explain their own reasoning — which will simplify the application-layer safety work that is currently required to compensate for alignment limitations.
Red-Teaming and Safety Testing Your Own Applications
Understanding how your model was aligned tells you where the safety boundaries are by design. But your application layer introduces new attack surfaces — prompt injection vectors, edge cases in your specific domain, and user populations with different intentions from the general consumer market the model was aligned for. Red-teaming your own application means attempting to elicit harmful or policy-violating outputs through adversarial inputs: direct instruction override attempts, roleplay scenarios that try to bypass safety training, edge case queries that probe the boundaries of your domain. Document every successful attack and implement application-layer mitigations — output filtering, input screening, human review triggers — for the failure modes that the model’s alignment training does not reliably catch for your specific context. This application-layer safety work is not optional for any LLM system deployed at scale — it is the difference between responsible deployment and deploying a system whose behaviour under adversarial conditions you have not tested.