XGBoost Early Stopping: How to Use It and Why It Matters

by

Early stopping is one of the most useful features in XGBoost — and one of the most frequently misconfigured. It prevents overfitting by halting training when the validation metric stops improving, and it eliminates the need to manually search for the right number of trees. This guide covers how to use it correctly, the gotchas that cause silent failures, and how to use the final model for prediction.

The Problem Early Stopping Solves

Without early stopping, you have to guess how many trees to train. Too few and the model underfits; too many and it memorises the training set. The relationship between tree count and validation performance follows the same curve for every boosting model: improvement early, plateau in the middle, degradation as overfitting sets in. Early stopping automates finding the peak of that curve. Set n_estimators to a generous ceiling (1000–5000), let early stopping find the actual optimum, and use model.best_iteration to know where it stopped.

Basic Early Stopping

import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(
    n_estimators=1000,
    learning_rate=0.05,
    max_depth=6,
    early_stopping_rounds=50,
    eval_metric='logloss',
    random_state=42,
    verbosity=1
)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=100
)
print(f"Best iteration: {model.best_iteration}")
print(f"Best score: {model.best_score}")
print(f"Accuracy: {accuracy_score(y_val, model.predict(X_val)):.4f}")

The eval_set Parameter Is Required

eval_set is required for early stopping to work — without it XGBoost has no validation metric to watch and early stopping silently does nothing. Pass a list of tuples: [(X_val, y_val)]. You can pass multiple sets; early stopping watches the last one. A common pattern is passing both train and val to monitor the gap:

model.fit(
    X_train, y_train,
    eval_set=[(X_train, y_train), (X_val, y_val)],
    verbose=100
)
# [0] train-logloss:0.651  validation_1-logloss:0.672
# Early stopping watches validation_1 (the last set)

Setting early_stopping_rounds

This controls how many consecutive rounds without improvement trigger a stop. Too low (5–10) causes premature stopping — the metric often plateaus briefly before improving again. Too high (300+) wastes training time. A practical rule tied to learning rate:

  • learning_rate=0.1 → 50 rounds
  • learning_rate=0.05 → 100 rounds
  • learning_rate=0.01 → 200 rounds

Lower learning rates make smaller improvements per round, so you need more patience before declaring convergence. When in doubt, err on the side of more rounds — the cost is a bit of extra training time, which is cheap compared to stopping too early and missing a better model.

Predicting with the Best Model

After early stopping, the model object remembers the best checkpoint. When you call predict(), it automatically uses the best iteration — you do not need to do anything special:

# This automatically uses model.best_iteration
preds = model.predict(X_val)
probas = model.predict_proba(X_val)[:, 1]

# Confirm which iteration is being used
print(f"Predicting with iteration {model.best_iteration} of {model.n_estimators}")

One common mistake: saving and reloading a model then calling predict without specifying iteration_range. When loaded from disk, the model defaults to using ALL trees. Fix:

model.save_model('xgb_model.json')
loaded = xgb.XGBClassifier()
loaded.load_model('xgb_model.json')

# Must specify best iteration when predicting after reload
preds = loaded.predict(X_val, iteration_range=(0, model.best_iteration + 1))

eval_metric Options

XGBoost supports many evaluation metrics for early stopping. The right one depends on your task:

  • Classification: logloss (default for binary), mlogloss (multiclass), auc, error (1 – accuracy)
  • Regression: rmse (default), mae, mape
  • Ranking: ndcg, map

Use auc for imbalanced classification — accuracy-based metrics are misleading when classes are skewed. Use logloss when you need calibrated probability outputs. You can pass a list to monitor multiple metrics simultaneously; early stopping watches the last one:

model = xgb.XGBClassifier(
    n_estimators=1000,
    early_stopping_rounds=100,
    eval_metric=['logloss', 'auc'],  # watches auc (last one)
    random_state=42, verbosity=0
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])

Early Stopping with Cross-Validation

Using early stopping with cross-validation is trickier because each fold has a different optimal number of trees. The standard approach: run cross-validation first to find the average best iteration, then retrain on the full training set for that many trees.

from sklearn.model_selection import StratifiedKFold
import numpy as np

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
best_iterations = []

for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):
    X_tr, X_vl = X.iloc[train_idx], X.iloc[val_idx]
    y_tr, y_vl = y.iloc[train_idx], y.iloc[val_idx]

    m = xgb.XGBClassifier(
        n_estimators=2000, learning_rate=0.05, max_depth=6,
        early_stopping_rounds=100, eval_metric='logloss',
        random_state=42, verbosity=0
    )
    m.fit(X_tr, y_tr, eval_set=[(X_vl, y_vl)])
    best_iterations.append(m.best_iteration)
    print(f"Fold {fold+1}: best_iteration={m.best_iteration}")

avg_best = int(np.mean(best_iterations))
print(f"Average best iteration: {avg_best}")

# Retrain on full data with the average best iteration
final_model = xgb.XGBClassifier(
    n_estimators=avg_best,
    learning_rate=0.05, max_depth=6,
    random_state=42, verbosity=0
)
final_model.fit(X, y)  # full dataset, no eval_set needed

Early Stopping in the Native XGBoost API

The native (non-sklearn) API gives more control, especially useful for custom objectives or when you want to access the full evaluation log:

import xgboost as xgb

dtrain = xgb.DMatrix(X_train, label=y_train)
dval = xgb.DMatrix(X_val, label=y_val)

params = {
    'objective': 'binary:logistic',
    'eval_metric': 'logloss',
    'learning_rate': 0.05,
    'max_depth': 6,
    'seed': 42,
    'verbosity': 0
}

evals_result = {}
booster = xgb.train(
    params,
    dtrain,
    num_boost_round=2000,
    evals=[(dtrain, 'train'), (dval, 'val')],
    early_stopping_rounds=100,
    evals_result=evals_result,
    verbose_eval=200
)

print(f"Best iteration: {booster.best_iteration}")
print(f"Best score: {booster.best_score}")

# Access full evaluation history
val_logloss = evals_result['val']['logloss']
print(f"Final val logloss: {val_logloss[booster.best_iteration]:.4f}")

Visualising the Training Curve

import matplotlib.pyplot as plt

# Using sklearn API — access eval results after fit
results = model.evals_result()
train_loss = results['validation_0']['logloss']
# If you passed two eval sets:
# val_loss = results['validation_1']['logloss']

epochs = range(len(train_loss))
plt.figure(figsize=(10, 4))
plt.plot(epochs, train_loss, label='Train logloss')
plt.axvline(x=model.best_iteration, color='red', linestyle='--', label=f'Best iteration ({model.best_iteration})')
plt.xlabel('Boosting round'); plt.ylabel('Log loss')
plt.title('XGBoost Training Curve with Early Stopping')
plt.legend(); plt.tight_layout(); plt.show()

Figure 1 — Early stopping behaviour: validation loss plateaus, then training stops

Training vs Validation Loss best_iteration Train Val Boosting rounds Loss

Common Mistakes

The mistakes that cause early stopping to silently not work or produce wrong results. Not passing eval_set — without it early stopping does nothing. Calling predict() on a reloaded model without specifying iteration_range — the loaded model uses all trees by default, not the best checkpoint. Setting early_stopping_rounds too low with a small learning rate — a patience of 20 rounds with learning_rate=0.01 will almost certainly stop too early. Using n_estimators as the final model’s tree count after cross-validation — always use the average best_iteration from folds, not the ceiling value. Forgetting that XGBoost 2.0 changed the early_stopping_rounds parameter location — in XGBoost 2.x it goes in the constructor, not in fit(); older code that passes it to fit() will raise a warning.

Early Stopping in a Full Pipeline

import optuna

def objective(trial):
    params = {
        'n_estimators': 2000,  # ceiling
        'learning_rate': trial.suggest_float('lr', 0.01, 0.3, log=True),
        'max_depth': trial.suggest_int('max_depth', 3, 9),
        'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-3, 10, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-3, 10, log=True),
        'early_stopping_rounds': 100,
        'eval_metric': 'logloss',
        'random_state': 42, 'verbosity': 0
    }
    model = xgb.XGBClassifier(**params)
    model.fit(X_train, y_train, eval_set=[(X_val, y_val)])
    return model.best_score  # logloss — Optuna will minimise

study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=50)
print(study.best_params)
print(f"Best logloss: {study.best_value:.4f}")

Using early stopping inside Optuna trials means every trial automatically finds its own optimal tree count. This prevents a confounded search where a good set of hyperparameters gets a bad score just because n_estimators is fixed at the wrong value. The combination of early stopping + Optuna is the most reliable way to tune XGBoost — each trial is evaluated at its own best checkpoint, not at a fixed round count that may be too high or too low for that hyperparameter combination.

Early Stopping for Regression

from xgboost import XGBRegressor
from sklearn.metrics import mean_squared_error
import numpy as np

reg = XGBRegressor(
    n_estimators=2000,
    learning_rate=0.05,
    max_depth=6,
    early_stopping_rounds=100,
    eval_metric='rmse',
    random_state=42,
    verbosity=0
)
reg.fit(X_train, y_train, eval_set=[(X_val, y_val)])
rmse = np.sqrt(mean_squared_error(y_val, reg.predict(X_val)))
print(f"Best iteration: {reg.best_iteration}")
print(f"Test RMSE: {rmse:.4f}")

Early Stopping with GPU Training

Early stopping works identically with GPU training — just add device='cuda' (XGBoost 2.x) or tree_method='gpu_hist' (XGBoost 1.x). The best_iteration checkpoint and prediction behaviour are unchanged. GPU training is significantly faster per round, which makes early stopping even more valuable: you can afford a generous ceiling like 5000 trees and a high patience like 200 rounds because the wall-clock cost of those extra rounds is low.

# XGBoost 2.x
model = xgb.XGBClassifier(
    n_estimators=5000,
    device='cuda',
    early_stopping_rounds=200,
    eval_metric='logloss',
    learning_rate=0.01,
    random_state=42, verbosity=0
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])

Why Not Just Use a Fixed n_estimators

The argument for fixing n_estimators is reproducibility — the same code always trains for the same number of rounds. But reproducibility and optimality rarely overlap. The right number of trees is a function of every other hyperparameter: change learning_rate from 0.05 to 0.01 and the optimal tree count roughly quintuples. Fix it at 500 with learning_rate=0.01 and your model is severely undertrained. Fix it at 500 with learning_rate=0.3 and it is probably overtrained. Early stopping dynamically adapts to whatever other hyperparameters you set, which is exactly what you want during tuning. The only valid reason to fix tree count is for a final production model where you want bitwise-identical outputs across runs — and even then, the correct approach is to run early stopping once, record best_iteration, then hardcode that specific value.

Checking for Data Leakage in the Validation Set

Early stopping is only as good as the validation set it watches. If your validation set has any temporal overlap with the training set — for example, you split a time series randomly rather than chronologically — the early stopping metric is optimistic and the model will overfit relative to what you see in production. Always split time series data chronologically: everything before date D goes to train, everything after goes to validation. For non-temporal data, stratified random splitting is fine. The early stopping score should be treated as a proxy for generalisation, not a guarantee — always reserve a held-out test set that is never seen by early stopping or hyperparameter search.

Saving the Best Model Correctly

When serialising a model trained with early stopping, include the best iteration in the filename or metadata so you can restore it correctly later. The safest pattern for production use:

import json, joblib

# Save model and best iteration together
model.save_model('xgb_model.json')
meta = {'best_iteration': model.best_iteration, 'best_score': model.best_score}
with open('xgb_meta.json', 'w') as f:
    json.dump(meta, f)

# Reload and predict correctly
loaded = xgb.XGBClassifier()
loaded.load_model('xgb_model.json')
with open('xgb_meta.json') as f:
    meta = json.load(f)

preds = loaded.predict(X_new, iteration_range=(0, meta['best_iteration'] + 1))
probas = loaded.predict_proba(X_new, iteration_range=(0, meta['best_iteration'] + 1))[:,1]

Storing the best iteration alongside the model file is a small discipline that prevents a subtle production bug: a model that appears to load and predict correctly but is secretly using 500 trees instead of the best 312, producing slightly worse predictions with no error or warning. Saving it explicitly makes the intent clear and auditable. For teams using MLflow or similar experiment trackers, log best_iteration and best_score as run metrics alongside the model artifact — this makes it easy to compare runs and verify that early stopping is working as expected across different hyperparameter configurations and dataset versions.

One final note on monitoring: the eval log printed during training (every N rounds via verbose) is your fastest sanity check that early stopping is configured correctly. If you see the validation metric flat-lining from round 1, early stopping is not receiving a useful signal — check that eval_set is passed and the metric direction is correct (lower is better for loss metrics, higher for AUC). If the metric improves monotonically all the way to n_estimators without stopping, your patience is too high or your ceiling is too low — increase the ceiling.

Early stopping is one of those techniques that costs almost nothing to add and consistently produces better models than a fixed tree count. The only reason not to use it is if you need perfectly reproducible tree counts across runs — in that case, find the best iteration with early stopping once, record it, then retrain with that fixed count. For everything else, use early stopping by default. Combined with a generous n_estimators ceiling and a patience value matched to your learning rate, it removes one of the most tedious hyperparameters to tune manually and lets you focus on the parameters that actually drive model quality.

Leave a Comment