XGBoost handles multiclass classification with the same API as binary classification — just change the objective and let XGBoost figure out the rest. The model trains one set of trees per class (one-vs-all internally), outputs a probability for each class, and predicts the class with the highest probability. This guide covers the full workflow: training, evaluation, class weighting for imbalanced multiclass problems, and interpreting predictions.
Basic Multiclass Setup
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
from sklearn.preprocessing import LabelEncoder
# Encode string labels to integers if needed
le = LabelEncoder()
y_encoded = le.fit_transform(y) # e.g. ['cat','dog','bird'] -> [0,1,2]
X_train, X_val, y_train, y_val = train_test_split(
X, y_encoded, test_size=0.2, stratify=y_encoded, random_state=42
)
model = xgb.XGBClassifier(
objective='multi:softprob', # outputs probability per class
num_class=len(le.classes_), # required: number of classes
n_estimators=500,
learning_rate=0.05,
max_depth=6,
eval_metric='mlogloss',
early_stopping_rounds=50,
random_state=42,
verbosity=0
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=100)
# Predict class labels
preds = model.predict(X_val)
print(f"Accuracy: {accuracy_score(y_val, preds):.4f}")
# Map back to original class names
pred_labels = le.inverse_transform(preds)
print(classification_report(y_val, preds, target_names=le.classes_))softmax vs softprob
XGBoost offers two multiclass objectives. multi:softmax outputs the predicted class label directly (integer). multi:softprob outputs a probability for each class — shape (n_samples, n_classes). Use softprob unless you only need hard labels and want to skip the probability computation. With either objective, model.predict() returns integer class labels and model.predict_proba() returns class probabilities.
# Probability output — shape (n_samples, n_classes)
probas = model.predict_proba(X_val)
print(f"Shape: {probas.shape}") # (n_val, n_classes)
print(f"Row sums (should be ~1.0): {probas.sum(axis=1)[:5]}")
# Confidence of top prediction
max_proba = probas.max(axis=1)
pred_class = probas.argmax(axis=1)
print(f"High confidence predictions (>0.9): {(max_proba > 0.9).sum()}")Multiclass Evaluation Metrics
Accuracy alone is insufficient for multiclass problems with class imbalance. Use macro-averaged F1 (treats all classes equally regardless of frequency) or weighted F1 (weights by class frequency). Confusion matrices show where the model confuses specific class pairs.
from sklearn.metrics import (
classification_report, confusion_matrix,
f1_score, roc_auc_score
)
import pandas as pd, seaborn as sns, matplotlib.pyplot as plt
# Per-class metrics
print(classification_report(y_val, preds, target_names=le.classes_))
# F1 scores
print(f"Macro F1: {f1_score(y_val, preds, average='macro'):.4f}")
print(f"Weighted F1: {f1_score(y_val, preds, average='weighted'):.4f}")
# Confusion matrix
cm = confusion_matrix(y_val, preds)
sns.heatmap(pd.DataFrame(cm, index=le.classes_, columns=le.classes_),
annot=True, fmt='d', cmap='Blues')
plt.title('Confusion Matrix'); plt.show()
# Multiclass AUC (one-vs-rest)
auc = roc_auc_score(y_val, probas, multi_class='ovr', average='macro')
print(f"Macro AUC (OvR): {auc:.4f}")Handling Imbalanced Multiclass Data
For imbalanced multiclass problems, assign sample weights proportional to the inverse of each class frequency. XGBoost does not have a scale_pos_weight equivalent for multiclass — use sample_weight in the fit() call instead.
import numpy as np
from sklearn.utils.class_weight import compute_sample_weight
# Compute sample weights — rare classes get higher weight
sample_weights = compute_sample_weight(class_weight='balanced', y=y_train)
model.fit(
X_train, y_train,
sample_weight=sample_weights,
eval_set=[(X_val, y_val)],
verbose=False
)
# Evaluate per-class recall to check if minority classes improved
print(classification_report(y_val, model.predict(X_val), target_names=le.classes_))Hyperparameter Tuning for Multiclass
import optuna
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
def objective(trial):
params = {
'objective': 'multi:softprob',
'num_class': len(le.classes_),
'n_estimators': trial.suggest_int('n_estimators', 200, 1500),
'learning_rate': trial.suggest_float('lr', 0.01, 0.3, log=True),
'max_depth': trial.suggest_int('max_depth', 3, 9),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-2, 10, log=True),
'eval_metric': 'mlogloss',
'random_state': 42, 'verbosity': 0
}
model = xgb.XGBClassifier(**params)
return cross_val_score(model, X_train, y_train, cv=cv, scoring='f1_macro').mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)
print(study.best_params)SHAP for Multiclass Explanations
For multiclass models, SHAP returns a list of arrays — one per class. Each array explains that class’s output score. Use the array for the predicted class for per-prediction explanations.
import shap
explainer = shap.TreeExplainer(model)
shap_values_mc = explainer.shap_values(X_val)
# List of n_classes arrays, each shape (n_val, n_features)
# Summary plot for each class
for i, class_name in enumerate(le.classes_):
print(f"
Class: {class_name}")
shap.summary_plot(shap_values_mc[i], X_val, plot_type='bar',
max_display=10, show=False)
plt.title(f'Feature importance — {class_name}')
plt.tight_layout(); plt.show()
# Explain a single prediction (predicted class)
row_idx = 0
pred_class_idx = model.predict(X_val.iloc[[row_idx]])[0]
shap.force_plot(
explainer.expected_value[pred_class_idx],
shap_values_mc[pred_class_idx][row_idx],
X_val.iloc[row_idx],
feature_names=X_val.columns.tolist()
)Figure 1 — Multiclass XGBoost: one set of trees per class, softmax over outputs
Label Encoding Pitfalls
XGBoost requires integer class labels starting from 0 for multiclass. String labels will raise an error. A few specifics to watch for. If your labels are integers but do not start from 0 (e.g., {1, 2, 3} or {10, 20, 30}), map them explicitly — XGBoost will interpret label 3 as “class index 3” which requires num_class=4 even if you only have 3 classes. If you use LabelEncoder, keep the fitted encoder alongside the model so you can decode predictions at inference. If some classes appear in validation but not training, the model has no trees for them and will always predict low probability — handle rare classes before training, not after.
One-vs-Rest vs Multinomial Training
XGBoost’s multiclass implementation is multinomial (all classes trained jointly in each round), not strictly one-vs-rest. This is more statistically efficient than training K separate binary models because tree splits can leverage information about all class boundaries simultaneously. The practical implication: multiclass XGBoost with 10 classes is roughly 10x slower to train than a binary classifier on the same dataset, because each boosting round trains 10 tree sets. Plan your Optuna trial budgets accordingly — 100 trials for binary may need to drop to 20-30 trials for large multiclass problems to finish in a reasonable time.
Predicting with Top-K Confidence Filtering
import numpy as np
def predict_with_confidence(model, X, threshold=0.6):
"""Return predictions only when model confidence exceeds threshold."""
probas = model.predict_proba(X)
max_proba = probas.max(axis=1)
pred_class = probas.argmax(axis=1)
results = []
for i, (cls, conf) in enumerate(zip(pred_class, max_proba)):
if conf >= threshold:
results.append({'class': le.inverse_transform([cls])[0], 'confidence': conf})
else:
results.append({'class': 'uncertain', 'confidence': conf})
return results
predictions = predict_with_confidence(model, X_val, threshold=0.7)
uncertain = sum(1 for p in predictions if p['class'] == 'uncertain')
print(f"Uncertain predictions: {uncertain}/{len(predictions)} ({uncertain/len(predictions)*100:.1f}%)")XGBoost multiclass is as straightforward to use as binary classification — the API changes are minimal (objective, num_class, eval_metric). The work is in evaluation: per-class F1, confusion matrices, and SHAP per class give you a complete picture of what the model is doing and where it is confused. For imbalanced multiclass, sample weights are the cleanest solution. For large class counts (20+), watch training time and reduce Optuna trial budgets accordingly. The multinomial training approach means the model sees all class boundaries simultaneously, which generally produces better calibrated multiclass probabilities than a stack of binary classifiers.