XGBoost handles multiclass classification with the same API as binary classification — just change the objective and let XGBoost figure out the rest. The model trains one set of trees per class (one-vs-all internally), outputs a probability for each class, and predicts the class with the highest probability. This guide covers the full workflow: training, evaluation, class weighting for imbalanced multiclass problems, and interpreting predictions.
Basic Multiclass Setup
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
from sklearn.preprocessing import LabelEncoder
# Encode string labels to integers if needed
le = LabelEncoder()
y_encoded = le.fit_transform(y) # e.g. ['cat','dog','bird'] -> [0,1,2]
X_train, X_val, y_train, y_val = train_test_split(
X, y_encoded, test_size=0.2, stratify=y_encoded, random_state=42
)
model = xgb.XGBClassifier(
objective='multi:softprob', # outputs probability per class
num_class=len(le.classes_), # required: number of classes
n_estimators=500,
learning_rate=0.05,
max_depth=6,
eval_metric='mlogloss',
early_stopping_rounds=50,
random_state=42,
verbosity=0
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=100)
# Predict class labels
preds = model.predict(X_val)
print(f"Accuracy: {accuracy_score(y_val, preds):.4f}")
# Map back to original class names
pred_labels = le.inverse_transform(preds)
print(classification_report(y_val, preds, target_names=le.classes_))softmax vs softprob
XGBoost offers two multiclass objectives. multi:softmax outputs the predicted class label directly (integer). multi:softprob outputs a probability for each class — shape (n_samples, n_classes). Use softprob unless you only need hard labels and want to skip the probability computation. With either objective, model.predict() returns integer class labels and model.predict_proba() returns class probabilities.
# Probability output — shape (n_samples, n_classes)
probas = model.predict_proba(X_val)
print(f"Shape: {probas.shape}") # (n_val, n_classes)
print(f"Row sums (should be ~1.0): {probas.sum(axis=1)[:5]}")
# Confidence of top prediction
max_proba = probas.max(axis=1)
pred_class = probas.argmax(axis=1)
print(f"High confidence predictions (>0.9): {(max_proba > 0.9).sum()}")Multiclass Evaluation Metrics
Accuracy alone is insufficient for multiclass problems with class imbalance. Use macro-averaged F1 (treats all classes equally regardless of frequency) or weighted F1 (weights by class frequency). Confusion matrices show where the model confuses specific class pairs.
from sklearn.metrics import (
classification_report, confusion_matrix,
f1_score, roc_auc_score
)
import pandas as pd, seaborn as sns, matplotlib.pyplot as plt
# Per-class metrics
print(classification_report(y_val, preds, target_names=le.classes_))
# F1 scores
print(f"Macro F1: {f1_score(y_val, preds, average='macro'):.4f}")
print(f"Weighted F1: {f1_score(y_val, preds, average='weighted'):.4f}")
# Confusion matrix
cm = confusion_matrix(y_val, preds)
sns.heatmap(pd.DataFrame(cm, index=le.classes_, columns=le.classes_),
annot=True, fmt='d', cmap='Blues')
plt.title('Confusion Matrix'); plt.show()
# Multiclass AUC (one-vs-rest)
auc = roc_auc_score(y_val, probas, multi_class='ovr', average='macro')
print(f"Macro AUC (OvR): {auc:.4f}")Handling Imbalanced Multiclass Data
For imbalanced multiclass problems, assign sample weights proportional to the inverse of each class frequency. XGBoost does not have a scale_pos_weight equivalent for multiclass — use sample_weight in the fit() call instead.
import numpy as np
from sklearn.utils.class_weight import compute_sample_weight
# Compute sample weights — rare classes get higher weight
sample_weights = compute_sample_weight(class_weight='balanced', y=y_train)
model.fit(
X_train, y_train,
sample_weight=sample_weights,
eval_set=[(X_val, y_val)],
verbose=False
)
# Evaluate per-class recall to check if minority classes improved
print(classification_report(y_val, model.predict(X_val), target_names=le.classes_))Hyperparameter Tuning for Multiclass
import optuna
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
def objective(trial):
params = {
'objective': 'multi:softprob',
'num_class': len(le.classes_),
'n_estimators': trial.suggest_int('n_estimators', 200, 1500),
'learning_rate': trial.suggest_float('lr', 0.01, 0.3, log=True),
'max_depth': trial.suggest_int('max_depth', 3, 9),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-2, 10, log=True),
'eval_metric': 'mlogloss',
'random_state': 42, 'verbosity': 0
}
model = xgb.XGBClassifier(**params)
return cross_val_score(model, X_train, y_train, cv=cv, scoring='f1_macro').mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)
print(study.best_params)SHAP for Multiclass Explanations
For multiclass models, SHAP returns a list of arrays — one per class. Each array explains that class’s output score. Use the array for the predicted class for per-prediction explanations.
import shap
explainer = shap.TreeExplainer(model)
shap_values_mc = explainer.shap_values(X_val)
# List of n_classes arrays, each shape (n_val, n_features)
# Summary plot for each class
for i, class_name in enumerate(le.classes_):
print(f"
Class: {class_name}")
shap.summary_plot(shap_values_mc[i], X_val, plot_type='bar',
max_display=10, show=False)
plt.title(f'Feature importance — {class_name}')
plt.tight_layout(); plt.show()
# Explain a single prediction (predicted class)
row_idx = 0
pred_class_idx = model.predict(X_val.iloc[[row_idx]])[0]
shap.force_plot(
explainer.expected_value[pred_class_idx],
shap_values_mc[pred_class_idx][row_idx],
X_val.iloc[row_idx],
feature_names=X_val.columns.tolist()
)Figure 1 — Multiclass XGBoost: one set of trees per class, softmax over outputs
Label Encoding Pitfalls
XGBoost requires integer class labels starting from 0 for multiclass. String labels will raise an error. A few specifics to watch for. If your labels are integers but do not start from 0 (e.g., {1, 2, 3} or {10, 20, 30}), map them explicitly — XGBoost will interpret label 3 as “class index 3” which requires num_class=4 even if you only have 3 classes. If you use LabelEncoder, keep the fitted encoder alongside the model so you can decode predictions at inference. If some classes appear in validation but not training, the model has no trees for them and will always predict low probability — handle rare classes before training, not after.
One-vs-Rest vs Multinomial Training
XGBoost’s multiclass implementation is multinomial (all classes trained jointly in each round), not strictly one-vs-rest. This is more statistically efficient than training K separate binary models because tree splits can leverage information about all class boundaries simultaneously. The practical implication: multiclass XGBoost with 10 classes is roughly 10x slower to train than a binary classifier on the same dataset, because each boosting round trains 10 tree sets. Plan your Optuna trial budgets accordingly — 100 trials for binary may need to drop to 20-30 trials for large multiclass problems to finish in a reasonable time.
Predicting with Top-K Confidence Filtering
import numpy as np
def predict_with_confidence(model, X, threshold=0.6):
"""Return predictions only when model confidence exceeds threshold."""
probas = model.predict_proba(X)
max_proba = probas.max(axis=1)
pred_class = probas.argmax(axis=1)
results = []
for i, (cls, conf) in enumerate(zip(pred_class, max_proba)):
if conf >= threshold:
results.append({'class': le.inverse_transform([cls])[0], 'confidence': conf})
else:
results.append({'class': 'uncertain', 'confidence': conf})
return results
predictions = predict_with_confidence(model, X_val, threshold=0.7)
uncertain = sum(1 for p in predictions if p['class'] == 'uncertain')
print(f"Uncertain predictions: {uncertain}/{len(predictions)} ({uncertain/len(predictions)*100:.1f}%)")XGBoost multiclass is as straightforward to use as binary classification — the API changes are minimal (objective, num_class, eval_metric). The work is in evaluation: per-class F1, confusion matrices, and SHAP per class give you a complete picture of what the model is doing and where it is confused. For imbalanced multiclass, sample weights are the cleanest solution. For large class counts (20+), watch training time and reduce Optuna trial budgets accordingly. The multinomial training approach means the model sees all class boundaries simultaneously, which generally produces better calibrated multiclass probabilities than a stack of binary classifiers.
Handling Imbalanced Multiclass Data
For imbalanced multiclass problems, assign sample weights proportional to inverse class frequency. XGBoost does not have a scale_pos_weight equivalent for multiclass — use sample_weight in the fit() call instead.
from sklearn.utils.class_weight import compute_sample_weight
# Rare classes get higher weight automatically
sample_weights = compute_sample_weight(class_weight='balanced', y=y_train)
model.fit(
X_train, y_train,
sample_weight=sample_weights,
eval_set=[(X_val, y_val)],
verbose=False
)
print(classification_report(y_val, model.predict(X_val), target_names=le.classes_))Hyperparameter Tuning for Multiclass
import optuna
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
def objective(trial):
params = {
'objective': 'multi:softprob',
'num_class': len(le.classes_),
'n_estimators': trial.suggest_int('n_estimators', 200, 1500),
'learning_rate': trial.suggest_float('lr', 0.01, 0.3, log=True),
'max_depth': trial.suggest_int('max_depth', 3, 9),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('col', 0.5, 1.0),
'reg_lambda': trial.suggest_float('lam', 1e-2, 10, log=True),
'eval_metric': 'mlogloss',
'random_state': 42, 'verbosity': 0
}
m = xgb.XGBClassifier(**params)
return cross_val_score(m, X_train, y_train, cv=cv, scoring='f1_macro').mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)
print(study.best_params)Multiclass Evaluation Metrics
Accuracy alone is insufficient when classes are imbalanced. Use macro-averaged F1 (treats all classes equally regardless of frequency), weighted F1 (weights by class frequency), or per-class recall to find which classes the model struggles with. Multiclass AUC-ROC uses one-vs-rest scoring and gives a single number for overall discriminative ability.
from sklearn.metrics import f1_score, roc_auc_score, classification_report
preds = model.predict(X_val)
probas = model.predict_proba(X_val)
print(classification_report(y_val, preds, target_names=le.classes_))
print(f"Macro F1: {f1_score(y_val, preds, average='macro'):.4f}")
print(f"Weighted F1: {f1_score(y_val, preds, average='weighted'):.4f}")
print(f"Macro AUC: {roc_auc_score(y_val, probas, multi_class='ovr', average='macro'):.4f}")SHAP for Multiclass Explanations
For multiclass models, SHAP returns a list of arrays — one per class. Each array explains that class’s raw output score. Use the array for the predicted class to explain individual predictions.
import shap, matplotlib.pyplot as plt
explainer = shap.TreeExplainer(model)
shap_values_mc = explainer.shap_values(X_val)
# List of n_classes arrays, each shape (n_val, n_features)
# Summary plot per class — which features matter for each class
for i, class_name in enumerate(le.classes_):
shap.summary_plot(shap_values_mc[i], X_val, plot_type='bar',
max_display=10, show=False)
plt.title(f'Feature importance — {class_name}')
plt.tight_layout(); plt.show()
# Explain one row for its predicted class
row_idx = 0
pred_cls = model.predict(X_val.iloc[[row_idx]])[0]
shap.force_plot(
explainer.expected_value[pred_cls],
shap_values_mc[pred_cls][row_idx],
X_val.iloc[row_idx],
feature_names=X_val.columns.tolist()
)Comparing SHAP importance across classes often reveals interesting model behaviour: a feature that strongly pushes toward Class A may have little effect on Class B, even when both are well-predicted. This level of insight is impossible to get from accuracy or F1 scores alone, and frequently guides feature engineering — if two classes share the same top SHAP features with opposite signs, those features are doing most of the discrimination work and deserve more careful engineering.
Softmax vs Softprob: When Each Matters
multi:softmax outputs the predicted class label as an integer — no probability computation, slightly faster. multi:softprob outputs a probability vector for each sample, which is required for threshold tuning, calibration, confidence filtering, and AUC-ROC scoring. In practice, always use softprob — the overhead is negligible and probabilities are almost always useful downstream. The only reason to use softmax is in a latency-critical serving environment where every millisecond counts and probability outputs are genuinely unused.
Label Encoding Pitfalls
XGBoost requires integer class labels starting from 0. String labels raise an error. Watch for these specifics: if your labels are integers that do not start from 0 (e.g. {1,2,3}), map them explicitly — XGBoost treats label 3 as “class index 3”, requiring num_class=4 even with only 3 classes. Keep the fitted LabelEncoder alongside the model to decode predictions at inference. If some classes appear in validation but not in training, the model has no trees for them and will always assign low probability — address rare classes before training.
One-vs-Rest vs Multinomial Training
XGBoost’s multiclass implementation is multinomial — all classes are trained jointly in each boosting round, not as K separate binary models. This is more statistically efficient because splits can leverage information about all class boundaries simultaneously. The practical implication: multiclass XGBoost with 10 classes trains roughly 10x slower than binary classification on the same dataset, because each round trains 10 tree sets. Plan Optuna trial budgets accordingly — 100 trials for binary may need to drop to 20-30 for large multiclass problems.
Predicting with Confidence Filtering
import numpy as np
def predict_with_confidence(model, X, le, threshold=0.6):
probas = model.predict_proba(X)
max_proba = probas.max(axis=1)
pred_class = probas.argmax(axis=1)
results = []
for cls, conf in zip(pred_class, max_proba):
if conf >= threshold:
results.append({'class': le.inverse_transform([cls])[0], 'confidence': float(conf)})
else:
results.append({'class': 'uncertain', 'confidence': float(conf)})
return results
preds = predict_with_confidence(model, X_val, le, threshold=0.7)
uncertain = sum(1 for p in preds if p['class'] == 'uncertain')
print(f"Uncertain: {uncertain}/{len(preds)} ({uncertain/len(preds)*100:.1f}%)")Confidence filtering is especially useful for multiclass models in production — returning “uncertain” rather than a low-confidence label is often the right behaviour for downstream systems that can route uncertain cases to a human reviewer or a fallback model. Set the threshold based on the precision you need at that confidence level using a calibration curve on a held-out validation set.
Figure 1 — Multiclass XGBoost: one tree set per class, softmax over joint outputs
Confusion Matrix Analysis
After training a multiclass model, always plot the confusion matrix before moving to hyperparameter tuning. It immediately shows which class pairs are being confused, which is more actionable than aggregate F1. If Class A is frequently confused with Class B, those two classes likely have overlapping features — look at their SHAP distributions side by side. If one class is consistently mispredicted as everything, it is likely underrepresented in training data and needs more examples or higher sample weight. The confusion matrix also reveals asymmetric confusion patterns — Class A misclassified as B more often than B as A — which can indicate that one class has more discriminative features and the other needs better feature engineering.
XGBoost multiclass is as straightforward as binary — the API changes are minimal (objective, num_class, eval_metric). The work is in evaluation: per-class F1, confusion matrices, and SHAP per class give a complete picture of model behaviour and confusion patterns. For imbalanced multiclass, sample weights are the cleanest solution. For large class counts, watch training time and reduce Optuna trial budgets. The multinomial training approach means the model sees all class boundaries simultaneously, producing better calibrated multiclass probabilities than a stack of binary classifiers. A final practical note: for problems with a very large number of classes (50+), consider hierarchical classification — group related classes into meta-classes, train a coarse classifier first, then a fine classifier within each group. This reduces the per-round tree count and often improves accuracy on rare classes that get lost in a flat 50-class softmax, at the cost of a more complex training pipeline and inference path.