Imbalanced classification — fraud detection, medical diagnosis, churn prediction, rare event detection — is where default model settings fail most predictably. A model that predicts the majority class for every sample achieves 99% accuracy on a 1:99 dataset while being completely useless. LightGBM has several built-in mechanisms for imbalanced data, and combining them with the right evaluation metrics and decision threshold tuning produces reliable models on datasets where naive approaches fall apart.
Why Accuracy Fails on Imbalanced Data
On a dataset where 1% of samples are positive, a model that always predicts negative achieves 99% accuracy. This is why accuracy is the wrong metric for imbalanced problems — it does not penalise the model for ignoring the minority class entirely. Use AUC-ROC (measures discriminative ability across all thresholds), PR-AUC/Average Precision (more informative than ROC when the positive class is rare), F1 score (harmonic mean of precision and recall, requires choosing a threshold), or recall at a fixed precision (useful when false negatives are costly).
is_unbalance: Automatic Class Weighting
The simplest approach — let LightGBM handle it. Setting is_unbalance=True automatically weights the minority class by the inverse of its frequency. For a 1:99 ratio, the minority class gets weight 99 and the majority class gets weight 1. This makes the model penalise minority class errors proportionally more during training.
import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, average_precision_score
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=42)
model = lgb.LGBMClassifier(
n_estimators=1000,
learning_rate=0.05,
num_leaves=63,
is_unbalance=True, # automatic class weighting
metric='auc',
random_state=42,
verbose=-1
)
model.fit(X_train, y_train,
eval_set=[(X_val, y_val)],
callbacks=[lgb.early_stopping(100), lgb.log_evaluation(200)])
probas = model.predict_proba(X_val)[:, 1]
print(f"AUC-ROC: {roc_auc_score(y_val, probas):.4f}")
print(f"PR-AUC: {average_precision_score(y_val, probas):.4f}")scale_pos_weight: Manual Class Weighting
For more control, set scale_pos_weight to the ratio of negative to positive samples. This is the standard approach when is_unbalance produces suboptimal results — you can tune the ratio as a hyperparameter:
import numpy as np
# Compute class ratio
n_neg = (y_train == 0).sum()
n_pos = (y_train == 1).sum()
ratio = n_neg / n_pos
print(f"Class ratio (neg/pos): {ratio:.1f}")
model = lgb.LGBMClassifier(
n_estimators=1000,
learning_rate=0.05,
num_leaves=63,
scale_pos_weight=ratio, # or tune this as a hyperparameter
metric='auc',
random_state=42, verbose=-1
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)],
callbacks=[lgb.early_stopping(100)])Stratified Splitting and Cross-Validation
Always use stratified splits with imbalanced data — random splitting can produce folds with zero or near-zero positive samples, especially at severe imbalance ratios. Both train_test_split and StratifiedKFold handle this with stratify=y:
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
model_cv = lgb.LGBMClassifier(is_unbalance=True, metric='auc', verbose=-1)
# Score with AUC-ROC — more informative than accuracy for imbalanced data
auc_scores = cross_val_score(model_cv, X, y, cv=cv, scoring='roc_auc')
ap_scores = cross_val_score(model_cv, X, y, cv=cv, scoring='average_precision')
print(f"AUC-ROC: {auc_scores.mean():.4f} (+/- {auc_scores.std():.4f})")
print(f"PR-AUC: {ap_scores.mean():.4f} (+/- {ap_scores.std():.4f})")Decision Threshold Tuning
By default, LightGBM (like most classifiers) uses 0.5 as the decision threshold. For imbalanced data, the optimal threshold is almost never 0.5. Lowering it catches more positives at the cost of more false positives; raising it reduces false positives at the cost of missing more positives. The right threshold depends on the cost of each type of error in your application.
from sklearn.metrics import precision_recall_curve, f1_score
import numpy as np
probas = model.predict_proba(X_val)[:, 1]
precisions, recalls, thresholds = precision_recall_curve(y_val, probas)
# Find threshold maximising F1
f1_scores = 2 * precisions * recalls / (precisions + recalls + 1e-8)
best_idx = np.argmax(f1_scores[:-1])
best_threshold = thresholds[best_idx]
print(f"Best F1 threshold: {best_threshold:.4f}")
print(f"Precision: {precisions[best_idx]:.4f} Recall: {recalls[best_idx]:.4f}")
# Apply threshold
preds = (probas >= best_threshold).astype(int)
print(f"F1 at best threshold: {f1_score(y_val, preds):.4f}")Figure 1 — Precision-Recall curve: choose threshold based on your cost trade-off
SMOTE: Synthetic Minority Oversampling
SMOTE generates synthetic minority class samples by interpolating between existing minority samples, rather than simply duplicating them. Use it only on training data, never on validation or test data:
from imblearn.over_sampling import SMOTE
smote = SMOTE(random_state=42, k_neighbors=5)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)
print(f"Original: {y_train.value_counts().to_dict()}")
print(f"Resampled: {pd.Series(y_resampled).value_counts().to_dict()}")
model = lgb.LGBMClassifier(n_estimators=500, metric='auc', verbose=-1)
model.fit(X_resampled, y_resampled, eval_set=[(X_val, y_val)],
callbacks=[lgb.early_stopping(50)])SMOTE works well when the minority class is compact and the decision boundary is relatively smooth. It can hurt performance when minority samples form multiple distinct clusters (interpolating between clusters creates unrealistic synthetic samples) or when the feature space is high-dimensional (interpolation in high dimensions is unreliable). Always compare SMOTE against is_unbalance=True and class weighting on your specific dataset — the best approach varies.
Combining Approaches: The Recommended Stack
The most reliable strategy for imbalanced LightGBM: use is_unbalance=True as a baseline, evaluate with AUC-ROC and PR-AUC, tune the decision threshold based on your precision-recall trade-off requirement, and use stratified cross-validation throughout. Add SMOTE only if the baseline underperforms on the minority class and you have verified it helps on held-out data. Tune scale_pos_weight with Optuna if the automatic weighting from is_unbalance is suboptimal. The combination of class weighting, the right metric, and threshold tuning resolves most imbalanced classification problems without needing complex sampling pipelines.
Choosing the Right Evaluation Metric
The metric you optimise during training and the metric you report should both reflect what matters for your application. AUC-ROC measures how well the model separates classes across all decision thresholds — it is threshold-independent and a good default for most imbalanced problems. However, AUC-ROC can be misleadingly optimistic on severely imbalanced datasets because it includes many threshold values where the model correctly classifies the abundant negative class. PR-AUC (area under the precision-recall curve) is more informative for severe imbalance — it focuses only on the positive class performance and is harder to inflate through correct negative predictions. For operational decisions, F-beta score lets you weight recall more than precision (beta > 1) or vice versa (beta < 1) depending on whether false negatives or false positives are more costly in your domain.
from sklearn.metrics import fbeta_score, classification_report
# F2 score — weights recall twice as much as precision
# Use when false negatives (missed positives) are costly, e.g. fraud or disease
preds = (probas >= best_threshold).astype(int)
f2 = fbeta_score(y_val, preds, beta=2)
print(f"F2 score: {f2:.4f}")
# Full classification report
print(classification_report(y_val, preds, target_names=['negative', 'positive']))Undersampling the Majority Class
Instead of oversampling the minority class, you can undersample the majority class to create a balanced training set. Random undersampling is the simplest approach and surprisingly effective when you have a lot of majority class data — removing 90% of majority samples still leaves enough for the model to learn the decision boundary.
from imblearn.under_sampling import RandomUnderSampler
rus = RandomUnderSampler(sampling_strategy=0.5, random_state=42)
# sampling_strategy=0.5 means minority:majority = 1:2 after resampling
X_resampled, y_resampled = rus.fit_resample(X_train, y_train)
print(pd.Series(y_resampled).value_counts())Undersampling loses information from the majority class, which can hurt if majority class patterns help the model distinguish it from the minority class. The advantage is that training is much faster on the smaller dataset. A common hybrid — Random Undersampling + SMOTE — undersamples the majority class and oversamples the minority class to a moderate imbalance ratio rather than full balance, combining the speed benefit of undersampling with the information preservation of oversampling.
Optuna Tuning for Imbalanced LightGBM
import optuna
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
def objective(trial):
params = {
'n_estimators': trial.suggest_int('n_estimators', 200, 2000),
'learning_rate': trial.suggest_float('lr', 0.005, 0.3, log=True),
'num_leaves': trial.suggest_int('num_leaves', 20, 150),
'min_child_samples': trial.suggest_int('min_child_samples', 20, 500),
'scale_pos_weight': trial.suggest_float('spw', 1.0, n_neg/n_pos * 2),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'subsample_freq': 1,
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-2, 10, log=True),
'random_state': 42, 'verbose': -1
}
model = lgb.LGBMClassifier(**params)
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring='average_precision')
return scores.mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(study.best_params)Notice that scale_pos_weight is included as an Optuna parameter with a search range up to twice the natural ratio. The natural ratio (n_neg/n_pos) is the starting point but the optimal value depends on the cost structure of your problem — tuning it lets Optuna find the best trade-off between precision and recall for your specific dataset and metric.
Choosing the Right Evaluation Metric
The metric you optimise during training and the metric you report should both reflect what matters for your application. AUC-ROC measures how well the model separates classes across all thresholds — it is threshold-independent and a good default. However, it can be misleadingly optimistic on severely imbalanced datasets because it includes threshold values where the model correctly classifies the abundant negatives. PR-AUC focuses only on positive class performance and is harder to inflate. F-beta score lets you weight recall over precision (beta > 1) when false negatives are more costly, or precision over recall (beta < 1) when false positives matter more.
from sklearn.metrics import fbeta_score, classification_report, average_precision_score
probas = model.predict_proba(X_val)[:,1]
preds = (probas >= best_threshold).astype(int)
f2 = fbeta_score(y_val, preds, beta=2) # weights recall 2x over precision
pr_auc = average_precision_score(y_val, probas)
print(f"F2: {f2:.4f} PR-AUC: {pr_auc:.4f}")
print(classification_report(y_val, preds, target_names=['neg','pos']))Undersampling the Majority Class
Instead of oversampling the minority class, undersampling the majority creates a balanced training set. Random undersampling is surprisingly effective when you have abundant majority class data — removing 90% still leaves enough signal. The advantage is much faster training.
from imblearn.under_sampling import RandomUnderSampler
rus = RandomUnderSampler(sampling_strategy=0.5, random_state=42)
# 0.5 → minority:majority = 1:2 after resampling
X_res, y_res = rus.fit_resample(X_train, y_train)
print(pd.Series(y_res).value_counts())Undersampling loses information from the majority class. A practical hybrid: undersample the majority to a moderate ratio then apply SMOTE to bring the minority class up — this is faster than pure SMOTE and often scores comparably. Always compare all three approaches (class weighting, SMOTE, undersampling) on your held-out data before committing.
Optuna Tuning for Imbalanced LightGBM
Include scale_pos_weight as an Optuna parameter rather than fixing it at the natural class ratio. The optimal weight depends on the cost structure of your problem and Optuna will find the right trade-off automatically against the PR-AUC objective.
import optuna
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
n_neg, n_pos = (y_train==0).sum(), (y_train==1).sum()
def objective(trial):
params = {
'n_estimators': trial.suggest_int('n_estimators', 200, 2000),
'learning_rate': trial.suggest_float('lr', 0.005, 0.3, log=True),
'num_leaves': trial.suggest_int('num_leaves', 20, 150),
'min_child_samples': trial.suggest_int('min_child_samples', 20, 500),
'scale_pos_weight': trial.suggest_float('spw', 1.0, (n_neg/n_pos)*2),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'subsample_freq': 1,
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-2, 10, log=True),
'random_state': 42, 'verbose': -1
}
m = lgb.LGBMClassifier(**params)
return cross_val_score(m, X_train, y_train, cv=cv, scoring='average_precision').mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(study.best_params)Calibrating Probabilities
Class weighting improves the model’s discriminative ability but can distort the probability estimates — a model trained with is_unbalance=True on a 1:99 dataset will assign higher probabilities to the positive class than a naively calibrated model expects. If you need well-calibrated probabilities (for expected value calculations, risk scoring, or downstream models that consume these probabilities), use isotonic regression or Platt scaling to recalibrate after training.
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
import matplotlib.pyplot as plt
# Calibrate using cross-validation on training data
calibrated = CalibratedClassifierCV(model, cv=5, method='isotonic')
calibrated.fit(X_train, y_train)
calibrated_probas = calibrated.predict_proba(X_val)[:,1]
# Plot calibration curve
frac_pos, mean_pred = calibration_curve(y_val, calibrated_probas, n_bins=10)
plt.plot(mean_pred, frac_pos, 's-', label='Calibrated LightGBM')
plt.plot([0,1],[0,1],'k--', label='Perfectly calibrated')
plt.xlabel('Mean predicted probability'); plt.ylabel('Fraction of positives')
plt.legend(); plt.show()Calibration matters most when probabilities feed into business logic directly — expected profit calculations in fraud prevention, risk stratification in healthcare, churn intervention prioritisation. If you only need a ranked list (sort by probability, act on the top N), calibration is less important than discrimination, and you can skip this step. For everything else, a calibrated model is a more trustworthy model.
The SVG diagram above illustrates the PR curve trade-off — the best F1 threshold sits at the knee of the curve where precision and recall are most balanced, and different applications should pick their threshold based on where their cost trade-off falls on that curve.
Imbalanced classification with LightGBM is largely a problem of metrics and calibration rather than model architecture. The algorithm itself is capable — the challenge is telling it what you care about (via class weights and loss metric), measuring it correctly (AUC-ROC, PR-AUC rather than accuracy), and translating probabilities into decisions at the right threshold for your application. Get those three things right and LightGBM is one of the strongest off-the-shelf choices for imbalanced tabular classification tasks.