Cross-validation gives you a reliable estimate of how an XGBoost model will perform on unseen data. A single train/test split is noisy — performance varies significantly depending on which samples happen to end up in the test set. Cross-validation reduces that variance by averaging over multiple splits. This guide covers how to do it correctly with XGBoost, how to combine it with early stopping, and what the common mistakes look like.
StratifiedKFold: The Right Default
For classification, always use StratifiedKFold rather than plain KFold. Stratified splits preserve the class distribution in each fold — critical for imbalanced datasets where a random split can put all rare class samples in one fold. For regression, plain KFold is fine.
import xgboost as xgb
from sklearn.model_selection import StratifiedKFold, cross_val_score
import numpy as np
model = xgb.XGBClassifier(
n_estimators=500,
learning_rate=0.05,
max_depth=6,
subsample=0.8,
colsample_bytree=0.8,
eval_metric='logloss',
random_state=42,
verbosity=0
)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# cross_val_score runs the full pipeline per fold
auc_scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc', n_jobs=-1)
print(f"AUC: {auc_scores.mean():.4f} (+/- {auc_scores.std():.4f})")
print(f"Individual folds: {auc_scores.round(4)}")Multiple Metrics in One Pass
from sklearn.model_selection import cross_validate
results = cross_validate(
model, X, y, cv=cv,
scoring=['roc_auc', 'average_precision', 'f1'],
return_train_score=True,
n_jobs=-1
)
for metric in ['roc_auc', 'average_precision', 'f1']:
val_mean = results[f'test_{metric}'].mean()
train_mean = results[f'train_{metric}'].mean()
gap = train_mean - val_mean
print(f"{metric}: val={val_mean:.4f} train={train_mean:.4f} gap={gap:.4f}")The train/val gap is your overfitting signal. A gap above 5-10 points on AUC means the model is memorising training data rather than generalising. Increase min_child_weight, add regularisation (reg_lambda), or reduce max_depth and num_leaves. A gap near zero with both scores low means underfitting — increase model complexity or add features.
Early Stopping with Cross-Validation
Combining early stopping with cross-validation correctly is the most reliable way to tune XGBoost. The correct pattern: run CV with early stopping to find the average best iteration, then retrain the final model on all data for that many trees.
best_iterations = []
for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):
X_tr, X_vl = X.iloc[train_idx], X.iloc[val_idx]
y_tr, y_vl = y.iloc[train_idx], y.iloc[val_idx]
m = xgb.XGBClassifier(
n_estimators=2000, # high ceiling
learning_rate=0.05,
max_depth=6,
early_stopping_rounds=100,
eval_metric='logloss',
random_state=42, verbosity=0
)
m.fit(X_tr, y_tr, eval_set=[(X_vl, y_vl)])
best_iterations.append(m.best_iteration)
print(f"Fold {fold+1}: best_iteration={m.best_iteration}, AUC={m.best_score:.4f}")
avg_best = int(np.mean(best_iterations))
print(f"
Average best iteration: {avg_best}")
# Retrain on full dataset for production
final_model = xgb.XGBClassifier(
n_estimators=avg_best, # use average, not ceiling
learning_rate=0.05, max_depth=6,
random_state=42, verbosity=0
)
final_model.fit(X, y)Repeated Cross-Validation
For small datasets (under 5k rows), a single 5-fold CV can be noisy — fold assignment randomness significantly affects scores. Repeated CV runs the same K-fold multiple times with different shuffles and averages results, giving a more stable estimate at the cost of more training runs.
from sklearn.model_selection import RepeatedStratifiedKFold
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=42)
scores = cross_val_score(model, X, y, cv=rskf, scoring='roc_auc', n_jobs=-1)
print(f"Repeated CV AUC: {scores.mean():.4f} (+/- {scores.std():.4f})")
print(f"Total folds evaluated: {len(scores)}") # 15Nested Cross-Validation for Unbiased Evaluation
When hyperparameters are tuned using cross-validation, the CV score is optimistic — it has been used to make model selection decisions. Nested CV uses an outer loop for evaluation and an inner loop for hyperparameter tuning, giving an unbiased performance estimate. It is computationally expensive but the right approach when you need to report honest model performance.
from sklearn.model_selection import GridSearchCV
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
param_grid = {
'n_estimators': [200, 500],
'max_depth': [4, 6],
'learning_rate': [0.05, 0.1]
}
# Inner loop: tune hyperparameters
inner_search = GridSearchCV(model, param_grid, cv=inner_cv,
scoring='roc_auc', n_jobs=-1)
# Outer loop: evaluate the best inner model
nested_scores = cross_val_score(inner_search, X, y, cv=outer_cv,
scoring='roc_auc', n_jobs=-1)
print(f"Nested CV AUC: {nested_scores.mean():.4f} (+/- {nested_scores.std():.4f})")Figure 1 — StratifiedKFold: class distribution preserved across all folds
Cross-Validation with Optuna
import optuna
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
def objective(trial):
params = {
'n_estimators': trial.suggest_int('n_est', 100, 1000),
'learning_rate': trial.suggest_float('lr', 0.01, 0.3, log=True),
'max_depth': trial.suggest_int('max_depth', 3, 9),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('col', 0.5, 1.0),
'reg_lambda': trial.suggest_float('lam', 1e-2, 10, log=True),
'min_child_weight': trial.suggest_int('mcw', 1, 10),
'random_state': 42, 'verbosity': 0
}
m = xgb.XGBClassifier(**params)
return cross_val_score(m, X_train, y_train, cv=cv, scoring='roc_auc').mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(f"Best AUC: {study.best_value:.4f}")
print(study.best_params)Common Cross-Validation Mistakes
The most costly mistakes when cross-validating XGBoost. Fitting a scaler or imputer on all data before CV — this leaks preprocessing parameters from validation folds into training; always use a pipeline. Using random splits for time series data — always use TimeSeriesSplit. Using accuracy as the scoring metric for imbalanced data — use AUC-ROC or average_precision instead. Reporting cross-validation scores after hyperparameter tuning as unbiased estimates — they are not; use nested CV or a held-out test set. Setting n_jobs=1 — cross-validation is embarrassingly parallel; always set n_jobs=-1 unless debugging.
How Many Folds?
5-fold is the standard default — it balances bias (too few folds underestimates variance) against compute cost (too many folds multiplies training runs). For small datasets (under 1k rows), 10-fold or leave-one-out gives a less biased estimate at acceptable cost. For large datasets (over 100k rows), 3-fold is often sufficient and significantly faster. For very imbalanced data, more folds help ensure each fold has enough positive samples — 10-fold or stratified 5-fold with repeats are both reasonable. The standard error of the mean over k folds is your uncertainty estimate — if it is larger than the performance difference between two models, you cannot reliably distinguish them without more data or more folds.
Figure 1 — StratifiedKFold: class ratio preserved in every validation fold
KFold for Regression
from sklearn.model_selection import KFold, cross_val_score
import numpy as np
kf = KFold(n_splits=5, shuffle=True, random_state=42)
reg = xgb.XGBRegressor(
n_estimators=500, learning_rate=0.05, max_depth=6,
random_state=42, verbosity=0
)
rmse_scores = cross_val_score(reg, X, y, cv=kf,
scoring='neg_root_mean_squared_error')
print(f"RMSE: {-rmse_scores.mean():.4f} (+/- {rmse_scores.std():.4f})")CV Score Stability and Dataset Size
On small datasets (under 2k rows), 5-fold CV variance is high enough that fold assignment randomness can swing scores by several percentage points. Three strategies to improve stability: increase to 10-fold, use repeated CV (3 repeats × 5 folds = 15 total), or use leave-one-out CV for datasets under 500 rows. On large datasets (over 50k rows), 3-fold is often sufficient — each fold has enough samples that the estimate stabilises, and training time becomes the binding constraint rather than variance. The right number of folds is the minimum that makes your fold-to-fold variance small relative to the performance differences you care about.
Parallelising Cross-Validation
Set n_jobs=-1 in cross_val_score and cross_validate to use all CPU cores. Each fold trains independently and parallelises perfectly. Note that if XGBoost is also set to use multiple threads (nthread=-1), you can get into a thread oversubscription situation — more threads than cores — which slows things down. For parallel CV, either set XGBoost to 1 thread and parallelise at the CV level, or run CV serially and let XGBoost use all threads:
# Option 1: parallel folds, single-threaded XGBoost
model_single = xgb.XGBClassifier(nthread=1, verbosity=0)
scores = cross_val_score(model_single, X, y, cv=cv, scoring='roc_auc', n_jobs=-1)
# Option 2: serial folds, multi-threaded XGBoost (better for large datasets)
model_multi = xgb.XGBClassifier(nthread=-1, verbosity=0)
scores = cross_val_score(model_multi, X, y, cv=cv, scoring='roc_auc', n_jobs=1)For datasets under 50k rows, Option 1 is usually faster overall. For larger datasets where XGBoost benefits significantly from multi-threading, Option 2 is better. Benchmark both on your hardware to confirm.
Interpreting CV Score Variance
The standard deviation across folds is as informative as the mean. A low mean with high standard deviation (e.g. AUC 0.82 ± 0.09) means the model is sensitive to which samples are in which fold — usually a sign of a small dataset, high class imbalance, or a model that is too complex for the available data. A high mean with low standard deviation (e.g. AUC 0.88 ± 0.02) means the model generalises consistently and the CV estimate is reliable. When comparing two models, their CV score difference needs to be larger than about 2x the combined standard error to be meaningfully distinguishable — two models at 0.850 ± 0.015 and 0.855 ± 0.015 are statistically indistinguishable on 5-fold CV.
CV with Custom Scoring
For domain-specific metrics not in sklearn’s standard set, define a custom scorer using make_scorer. Custom scorers are useful when your operational metric differs from standard statistical ones — profit, prevented incidents, cost savings. Build cross-validation around what actually matters for your application. The difference between optimising for AUC and optimising for profit at a specific threshold can be significant, especially on imbalanced problems where AUC rewards discrimination across all thresholds equally rather than performance at the one threshold you will actually deploy.
Reporting CV Results Clearly
When reporting cross-validation results, always include: number of folds, whether stratification was used, the metric, the mean, the standard deviation, and whether results are from simple or nested CV. A complete report: “5-fold stratified CV, AUC-ROC 0.847 ± 0.023, hyperparameters selected by nested inner 3-fold CV.” This lets readers assess reliability and evaluation methodology soundness. Reporting only “AUC 0.85” without fold count, stratification, and hyperparameter tuning context leaves the estimate uninterpretable — it could be honest or heavily inflated depending on methodology choices that are not stated.
When a Single Validation Split Is Acceptable
Cross-validation is the default, but a single held-out split is acceptable in specific situations: when your dataset is very large (over 500k rows) and each fold would take hours to train, when you have a natural temporal split that makes CV inappropriate (time series where you cannot shuffle), or when the model is being evaluated on a genuinely held-out dataset that was never used for any training or tuning decision. In the last case, a single final evaluation on the test set is the right methodology — running multiple evaluations on the same test set and reporting the best inflates scores the same way as running multiple CV folds without proper nesting. The test set is a one-shot measurement, not a repeated metric to optimise against. For everything else where you have the compute budget, cross-validation produces more reliable estimates and should be the default.
Cross-validation is not a step to skip to save time — it is the only reliable way to know whether an XGBoost model will generalise. A single train/test split is a coin flip on which samples end up where. Five folds at 5x the compute cost of a single split is worth it every time. The patterns here — stratified splits, early stopping with averaged best iteration, nested CV for unbiased reporting, Optuna inside cross-validation — compose into a complete evaluation workflow that produces honest, stable performance estimates you can act on with confidence rather than hope. The patterns here — stratified splits, early stopping with averaged best iteration, nested CV for unbiased reporting, Optuna inside cross-validation — compose into a complete evaluation workflow that produces honest, stable performance estimates you can act on with confidence. That is a meaningful upgrade over a single train-test split, and it costs one line of code.