XGBoost’s GPU training can cut training time by 5–50x depending on dataset size and hardware. On a dataset with a million rows the difference between CPU and GPU training is often the difference between waiting 20 minutes per Optuna trial and waiting 45 seconds. This guide covers how to enable GPU training, what the parameter changes look like between XGBoost versions, how to benchmark the speedup on your own hardware, and what to expect when things do not work.
XGBoost 2.x vs 1.x: Two Different Parameter Names
The most common source of confusion with XGBoost GPU training is that the parameter changed between major versions. If you are copy-pasting from older tutorials, you will hit deprecation warnings or errors:
- XGBoost 1.x:
tree_method='gpu_hist' - XGBoost 2.x:
device='cuda'(recommended) ordevice='gpu'
Check your version first:
import xgboost as xgb
print(xgb.__version__) # 2.x vs 1.xEnabling GPU Training — XGBoost 2.x
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(
n_estimators=1000,
learning_rate=0.05,
max_depth=6,
device='cuda', # XGBoost 2.x GPU parameter
early_stopping_rounds=50,
eval_metric='logloss',
random_state=42,
verbosity=1
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=100)
print(f"Accuracy: {accuracy_score(y_val, model.predict(X_val)):.4f}")Enabling GPU Training — XGBoost 1.x
model = xgb.XGBClassifier(
n_estimators=1000,
learning_rate=0.05,
max_depth=6,
tree_method='gpu_hist', # XGBoost 1.x GPU parameter
gpu_id=0, # which GPU to use (default 0)
early_stopping_rounds=50,
eval_metric='logloss',
random_state=42
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=100)Multi-GPU Training
# XGBoost 2.x — specify multiple GPUs
model = xgb.XGBClassifier(
n_estimators=2000,
device='cuda',
n_jobs=-1, # use all available CPU threads for data loading
random_state=42
)
# XGBoost 1.x — use nthread for multi-GPU with Dask
# (multi-GPU is simpler in 2.x)
import dask.dataframe as dd
from dask.distributed import Client
client = Client(n_workers=2) # one worker per GPU
Selecting a Specific GPU
# XGBoost 2.x — target a specific GPU by index
model = xgb.XGBClassifier(device='cuda:1') # use GPU index 1
# Alternatively, set via environment variable before training
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '1'
model = xgb.XGBClassifier(device='cuda')When running multiple training jobs in parallel (e.g. parallel Optuna trials), set CUDA_VISIBLE_DEVICES differently for each process to avoid GPU memory conflicts. A common pattern is to assign GPU index = trial_number % num_gpus.
Benchmarking CPU vs GPU
import time
params_base = dict(n_estimators=500, learning_rate=0.05, max_depth=6,
eval_metric='logloss', random_state=42, verbosity=0)
# CPU
t0 = time.time()
m_cpu = xgb.XGBClassifier(**params_base, device='cpu')
m_cpu.fit(X_train, y_train, eval_set=[(X_val, y_val)])
cpu_time = time.time() - t0
# GPU
t0 = time.time()
m_gpu = xgb.XGBClassifier(**params_base, device='cuda')
m_gpu.fit(X_train, y_train, eval_set=[(X_val, y_val)])
gpu_time = time.time() - t0
print(f"CPU: {cpu_time:.1f}s GPU: {gpu_time:.1f}s Speedup: {cpu_time/gpu_time:.1f}x")Expected Speedups by Dataset Size
GPU training is not always faster — the speedup depends heavily on dataset size and tree depth. On small datasets (under 10k rows), data transfer overhead to the GPU dominates and CPU is often faster or equivalent. The break-even point is roughly 50k–100k rows for typical tabular data. At 1M+ rows with max_depth 6–10, GPU is usually 10–50x faster than a single CPU core, or 3–10x faster than all CPU cores combined.
Figure 1 — Approximate GPU speedup vs dataset size (RTX 3080, XGBoost 2.x)
GPU with the Native API
import xgboost as xgb
dtrain = xgb.DMatrix(X_train, label=y_train)
dval = xgb.DMatrix(X_val, label=y_val)
params = {
'objective': 'binary:logistic',
'eval_metric': 'logloss',
'device': 'cuda', # XGBoost 2.x
'learning_rate': 0.05,
'max_depth': 6,
'seed': 42
}
booster = xgb.train(
params, dtrain,
num_boost_round=2000,
evals=[(dval, 'val')],
early_stopping_rounds=100,
verbose_eval=200
)GPU Training in Google Colab
Colab’s free tier includes an NVIDIA T4 GPU. Enable it under Runtime → Change runtime type → GPU. XGBoost is pre-installed with GPU support — no extra installation needed:
# In Colab — GPU already available, just set device
import xgboost as xgb
print(xgb.__version__) # should be 2.x
model = xgb.XGBClassifier(device='cuda', n_estimators=1000, verbosity=1)
model.fit(X_train, y_train)
# Verify GPU is being used
# Training output will show: [GPU] ...
The T4 GPU in Colab’s free tier provides roughly 8–12x speedup over Colab’s CPU for large datasets. The Pro tier includes A100 access with significantly higher memory bandwidth, which benefits very large datasets and deep trees.
Memory Management
GPU memory (VRAM) limits how large a dataset you can train on in one go. XGBoost’s GPU implementation loads the entire dataset into VRAM — if your data exceeds GPU memory, training will fail with an out-of-memory error. Rough estimates for 6GB VRAM (RTX 3060/3080): up to ~5M rows with 50 float32 features. Strategies when data does not fit in VRAM:
- Use
device='cpu'for the full dataset and GPU only for hyperparameter search on a sample - Reduce feature count via selection before GPU training
- Use
max_binto reduce histogram memory:max_bin=128instead of the default 256 - Use XGBoost’s external memory mode:
xgb.DMatrix('data.libsvm#dtrain.cache')— slower but fits arbitrarily large data
Installing XGBoost with GPU Support
# Standard pip install includes GPU support (requires CUDA toolkit installed)
pip install xgboost
# Verify GPU is accessible
python -c "import xgboost as xgb; print(xgb.build_info())"
# Look for: USE_CUDA: 1
# If pip version lacks GPU, install from conda-forge
conda install -c conda-forge py-xgboost-gpu
# Check CUDA version compatibility
nvidia-smi # shows CUDA version
nvcc --version # shows toolkit versionTroubleshooting: GPU Not Being Used
Common reasons XGBoost silently falls back to CPU even when you set device='cuda': the XGBoost build was compiled without CUDA support (check xgb.build_info()), CUDA toolkit is not installed or the wrong version, the NVIDIA driver is outdated, or on Windows the CUDA DLLs are not in PATH. The clearest diagnostic is to check build_info() — if USE_CUDA is 0, your install does not have GPU support and you need to reinstall from conda-forge or the CUDA-enabled pip package. If it is 1 but training is still slow, check GPU utilisation with nvidia-smi dmon during training — utilisation should be 70–100% if GPU training is active.
GPU Training with Optuna
import optuna
def objective(trial):
params = {
'n_estimators': 2000,
'device': 'cuda',
'learning_rate': trial.suggest_float('lr', 0.01, 0.3, log=True),
'max_depth': trial.suggest_int('max_depth', 3, 10),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
'reg_alpha': trial.suggest_float('reg_alpha', 1e-3, 10, log=True),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-3, 10, log=True),
'early_stopping_rounds': 100,
'eval_metric': 'logloss',
'random_state': 42, 'verbosity': 0
}
model = xgb.XGBClassifier(**params)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])
return model.best_score
study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=100) # 100 GPU trials in minutes, not hoursThe combination of GPU training and Optuna is where the speedup pays off most. Running 100 Optuna trials on a large dataset takes hours on CPU and minutes on GPU. That difference changes what kind of search is practical — with GPU you can run 200 trials with 5-fold cross-validation per trial in the time CPU would spend on 20 trials with a single train/val split. That is a qualitatively different level of hyperparameter optimisation.
Apple Silicon: MPS Backend
XGBoost 2.x added experimental Metal Performance Shaders (MPS) support for Apple Silicon. Use device='mps' on M1/M2/M3 Macs. The speedup is more modest than NVIDIA CUDA — typically 2–4x over CPU — because Apple’s unified memory architecture means there is no PCIe transfer bottleneck but also no dedicated high-bandwidth VRAM. Still worth enabling on Apple hardware for longer training runs:
# Apple Silicon (M1/M2/M3) — experimental
model = xgb.XGBClassifier(device='mps', n_estimators=1000)
# Falls back to CPU if MPS is not availableGPU training in XGBoost is one of the lowest-effort, highest-impact performance upgrades available for tabular ML. On large datasets it makes the difference between waiting and not waiting — and that difference matters when you are running Optuna searches, cross-validation, or rapid prototyping cycles. The setup is one parameter change. The speedup is real. Check your XGBoost version, set device='cuda' (2.x) or tree_method='gpu_hist' (1.x), verify with build_info() that GPU support is compiled in, and benchmark against CPU on your actual dataset to confirm the gain before committing to a GPU workflow.
How XGBoost’s GPU Implementation Works
Understanding what the GPU actually does helps set realistic expectations. XGBoost’s GPU implementation accelerates the histogram construction step — the most computationally expensive part of tree building. For each feature, XGBoost bins the data into histograms and finds the best split by scanning histogram buckets. On CPU this is parallelised across cores; on GPU it is parallelised across thousands of CUDA cores simultaneously. The gain is largest when there are many features and many rows, because that is when histogram construction dominates training time. For shallow trees (max_depth 3–4) or small datasets where tree building finishes quickly, the GPU advantage shrinks because data transfer overhead becomes a larger fraction of total time. This is why you should always benchmark on your specific dataset rather than assuming a fixed speedup factor.
GPU Training Does Not Change Results
A common concern: does GPU training produce different models than CPU training? The short answer is no — for the same random seed and hyperparameters, GPU and CPU XGBoost produce numerically identical or near-identical models. Any differences are due to floating-point rounding order in parallel operations, which are negligibly small in practice. You can train with GPU for speed and deploy on CPU for inference without any accuracy degradation. This is a meaningful advantage over some deep learning frameworks where GPU training can introduce non-determinism.
When GPU Training Is Not Worth It
GPU training adds infrastructure complexity — you need a CUDA-capable GPU, the right driver and toolkit versions, and potentially cloud GPU instances. It is not worth the setup for small datasets where CPU is fast enough (training completes in under 2 minutes), for one-off models where you train once and deploy, or for environments where GPU access is expensive and your dataset is small enough that the time savings do not offset the cost. The sweet spot is iterative development on large datasets — data science workflows where you train many variants, run long Optuna searches, or do frequent retraining as data grows. If your XGBoost training takes more than 5 minutes on CPU, GPU is worth investigating.
Cloud GPU Options for XGBoost
If you do not have a local GPU, cloud options are practical for XGBoost training. Google Colab free tier (T4 GPU) is the lowest-friction option for experimentation — XGBoost with CUDA support is pre-installed and the T4 handles most datasets up to a few million rows comfortably. For production training pipelines, AWS SageMaker ml.p3.2xlarge (V100) or ml.g4dn.xlarge (T4) instances are cost-effective for periodic retraining jobs. For Optuna searches on very large datasets, a spot instance on an A10G or A100 GPU reduces search time from days to hours. The key is matching instance type to dataset size — a V100 for a 100k row dataset is wasteful; a T4 for a 50M row dataset will run out of VRAM.
Combining GPU Training with Feature Importance
import pandas as pd
# Feature importance works identically with GPU-trained models
model = xgb.XGBClassifier(device='cuda', n_estimators=1000, verbosity=0)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)],
callbacks=[xgb.callback.EarlyStopping(rounds=50)])
fi = pd.Series(model.feature_importances_, index=X_train.columns)
fi.sort_values(ascending=False).head(20).plot(kind='barh', figsize=(8,6))
# SHAP values also work — but SHAP computation runs on CPU
import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_val) # model transferred to CPU for SHAPSHAP value computation via the shap library always runs on CPU even for GPU-trained XGBoost models — the TreeExplainer transfers the model internally. This is fine for inference time, which is CPU-bound anyway. For training-time feature importance (not SHAP), you can call model.get_booster().get_fscore() which also runs on CPU after training completes. The GPU-trained model object is otherwise identical to a CPU-trained one and supports all the same methods and serialisation formats.