CatBoost Tutorial: A Practical Guide for Python Developers

by

CatBoost quietly became one of the best gradient boosting libraries available. It handles categorical features natively without preprocessing, trains fast on GPU, and consistently outperforms XGBoost and LightGBM on datasets with lots of categorical data — which describes most real-world tabular datasets. If you have been manually one-hot encoding categoricals before throwing them at XGBoost, this guide will save you a lot of work.

Installation

pip install catboost
# GPU support is included — no separate install needed
# Verify:
python -c "import catboost; print(catboost.__version__)"

Your First CatBoost Classifier

The minimal training loop uses the scikit-learn compatible API. CatBoost works with pandas DataFrames directly — no need to convert to numpy arrays first.

import pandas as pd
from catboost import CatBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load data
df = pd.read_csv('your_data.csv')
X = df.drop('target', axis=1)
y = df['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Identify categorical columns
cat_features = [col for col in X_train.columns if X_train[col].dtype == 'object']

model = CatBoostClassifier(
    iterations=500,
    learning_rate=0.05,
    depth=6,
    cat_features=cat_features,
    eval_metric='Accuracy',
    verbose=100,          # print progress every 100 rounds
    random_seed=42
)

model.fit(X_train, y_train, eval_set=(X_test, y_test))
preds = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, preds):.4f}")

The key parameter here is cat_features — a list of column names (or indices) that CatBoost should treat as categorical. Pass the raw string columns without any encoding and CatBoost handles the rest internally using ordered target statistics.

How CatBoost Handles Categorical Features

Most boosting libraries require you to encode categorical features before training — one-hot encoding, label encoding, or target encoding. CatBoost does this internally using a technique called ordered target statistics (also called ordered boosting). For each categorical value, it computes a target statistic (a form of target encoding) using only the data points that appeared before the current one in a random permutation of the training set. This prevents target leakage while still capturing the predictive signal in the categorical feature. The result is that raw string columns — “New York”, “London”, “Tokyo” — go in and CatBoost produces a competitive model without any preprocessing from you.

High-cardinality categoricals (hundreds or thousands of unique values) are where CatBoost particularly shines. One-hot encoding a column with 500 unique values adds 500 binary columns to your dataset and makes tree splits expensive. CatBoost encodes them into a single numeric representation per observation, keeping the dataset dense and training fast.

Handling Missing Values

CatBoost handles missing values natively for numeric features — you do not need to impute before training. For categorical features, replace NaN with a string like "MISSING" before training so CatBoost treats it as a category rather than skipping it:

X_train[cat_features] = X_train[cat_features].fillna('MISSING')
X_test[cat_features] = X_test[cat_features].fillna('MISSING')

Key Hyperparameters

CatBoost has sensible defaults that work reasonably well out of the box, but knowing the key levers helps when you need to improve performance or reduce overfitting:

  • iterations — number of trees. More is usually better up to a point; use early stopping (below) to find the right number automatically.
  • learning_rate — step size. Lower (0.01–0.05) with more iterations often beats higher with fewer. Default is auto-set based on iterations.
  • depth — tree depth. CatBoost uses symmetric trees (all branches at the same depth), so depth 6 is deeper than it sounds. Range 4–10; deeper trees overfit more.
  • l2_leaf_reg — L2 regularisation. Increase to reduce overfitting. Default is 3; try 1–10.
  • random_strength — randomness added when selecting split scores. Higher values add more regularisation via noise. Default 1.
  • bagging_temperature — controls Bayesian bootstrap intensity. Default 1; range 0 (no bagging) to higher values for more aggressive sampling.

Early Stopping

Early stopping is essential — it prevents overfitting and saves training time by stopping when the validation metric stops improving.

model = CatBoostClassifier(
    iterations=2000,       # set high — early stopping will cut this
    learning_rate=0.03,
    depth=6,
    cat_features=cat_features,
    early_stopping_rounds=50,  # stop if no improvement for 50 rounds
    eval_metric='Accuracy',
    verbose=200,
    random_seed=42
)

model.fit(
    X_train, y_train,
    eval_set=(X_test, y_test),
    use_best_model=True    # use the checkpoint with best eval score
)

print(f"Best iteration: {model.best_iteration_}")
print(f"Best score: {model.best_score_}")

CatBoost for Regression

from catboost import CatBoostRegressor

reg = CatBoostRegressor(
    iterations=1000,
    learning_rate=0.05,
    depth=6,
    cat_features=cat_features,
    loss_function='RMSE',
    eval_metric='RMSE',
    early_stopping_rounds=50,
    verbose=200,
    random_seed=42
)
reg.fit(X_train, y_train, eval_set=(X_test, y_test), use_best_model=True)

from sklearn.metrics import mean_squared_error
import numpy as np
rmse = np.sqrt(mean_squared_error(y_test, reg.predict(X_test)))
print(f"Test RMSE: {rmse:.4f}")

GPU Training

CatBoost’s GPU support is one of its strongest features — it is faster than XGBoost on GPU for most datasets, especially those with categorical features. Just add task_type="GPU":

model = CatBoostClassifier(
    iterations=1000,
    task_type="GPU",       # that's it
    devices='0',           # GPU index; '0:1' for multiple GPUs
    cat_features=cat_features,
    verbose=100
)
model.fit(X_train, y_train, eval_set=(X_test, y_test))

GPU training is 5-50x faster than CPU depending on dataset size, and the results are numerically identical. On Apple Silicon, use task_type="CPU" — Metal GPU is not supported, but CatBoost’s CPU implementation is still fast.

Feature Importance

import pandas as pd
import matplotlib.pyplot as plt

# Get feature importance
fi = pd.Series(model.get_feature_importance(), index=X_train.columns)
fi.sort_values(ascending=False).head(15).plot(kind='barh', figsize=(8,6))
plt.title('CatBoost Feature Importance')
plt.tight_layout()
plt.show()

# SHAP values for explainability
shap_values = model.get_feature_importance(
    data=catboost.Pool(X_test, label=y_test, cat_features=cat_features),
    type='ShapValues'
)
# shap_values shape: (n_samples, n_features + 1)
# Last column is the bias term; drop it
shap_values = shap_values[:, :-1]

Saving and Loading Models

# Save
model.save_model('catboost_model.cbm')

# Load
from catboost import CatBoostClassifier
loaded = CatBoostClassifier()
loaded.load_model('catboost_model.cbm')
preds = loaded.predict(X_test)

The .cbm format stores the full model including categorical feature metadata — you do not need to pass cat_features again at inference time when loading from a .cbm file.

CatBoost vs XGBoost vs LightGBM

Figure 1 — Boosting library trade-offs at a glance

Feature CatBoost XGBoost LightGBM Categorical features Native (no prep) Manual encoding Partial support Missing values Native (numeric) Native Native Training speed (CPU) Medium Medium Fastest Training speed (GPU) Fastest Fast Fast Best for Mixed/cat-heavy data General purpose Large datasets Sklearn compatible Yes Yes Yes

Using CatBoost with scikit-learn Pipelines

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from catboost import CatBoostClassifier

# CatBoost is sklearn-compatible — drop into any pipeline
# Note: cat_features are passed directly to CatBoost, not via the pipeline
cat_idx = [X_train.columns.get_loc(c) for c in cat_features]  # column indices

pipeline = Pipeline([
    # Add other transformers for numeric features here if needed
    ('model', CatBoostClassifier(
        iterations=500,
        cat_features=cat_idx,  # use indices when inside a pipeline
        verbose=0,
        random_seed=42
    ))
])

pipeline.fit(X_train, y_train)
print(pipeline.score(X_test, y_test))

Hyperparameter Search with Optuna

import optuna
from catboost import CatBoostClassifier
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        'iterations': trial.suggest_int('iterations', 200, 1000),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.2, log=True),
        'depth': trial.suggest_int('depth', 4, 10),
        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1, 10),
        'bagging_temperature': trial.suggest_float('bagging_temperature', 0, 1),
        'cat_features': cat_features,
        'verbose': 0,
        'random_seed': 42
    }
    model = CatBoostClassifier(**params)
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='accuracy')
    return scores.mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)
print("Best params:", study.best_params)
print("Best CV accuracy:", study.best_value)

CatBoost’s native categorical handling is its real differentiator. If your dataset has more than two or three categorical columns — user IDs, city names, product categories, device types — it is worth benchmarking CatBoost against your current approach. The setup is minimal, the defaults are reasonable, and on categorical-heavy datasets it frequently beats a heavily-tuned XGBoost or LightGBM without any feature engineering on your part. Start with the basic classifier, enable early stopping, and run Optuna for 50 trials — that workflow gets you to a strong baseline in under an hour.

Understanding Ordered Boosting

One reason CatBoost often outperforms other boosting libraries even on purely numeric datasets is its use of ordered boosting. Standard gradient boosting trains each tree on residuals from all previous trees using the same training data — this introduces a subtle overfitting bias because the model that produced the residuals was itself trained on the same data. CatBoost addresses this with a random permutation of the training data: when computing the residual for sample i, it uses a model trained only on samples that appeared before sample i in the permutation. This ordered approach reduces overfitting, which is especially noticeable on smaller datasets where standard boosting tends to overfit aggressively. The downside is slightly slower training compared to standard boosting — a trade-off that is worth understanding when benchmarking against XGBoost or LightGBM.

CatBoostPool for Efficient Data Handling

For large datasets, loading data into a CatBoost Pool object upfront is more memory-efficient than passing DataFrames directly, because the Pool handles categorical encoding once during construction rather than on every training iteration:

from catboost import Pool

# Build pools once — more efficient for large datasets
train_pool = Pool(X_train, label=y_train, cat_features=cat_features)
test_pool = Pool(X_test, label=y_test, cat_features=cat_features)

model = CatBoostClassifier(iterations=500, verbose=100)
model.fit(train_pool, eval_set=test_pool, use_best_model=True)

# Predict
preds = model.predict(test_pool)
probas = model.predict_proba(test_pool)[:, 1]  # probability of positive class

Multiclass Classification

model = CatBoostClassifier(
    iterations=500,
    loss_function='MultiClass',     # automatically handles N classes
    eval_metric='Accuracy',
    cat_features=cat_features,
    verbose=100
)
model.fit(X_train, y_train, eval_set=(X_test, y_test), use_best_model=True)

# Probabilities for each class
probas = model.predict_proba(X_test)  # shape: (n_samples, n_classes)
classes = model.classes_              # class labels in order

Common Pitfalls

A few mistakes that come up frequently when starting with CatBoost. First, passing integer-encoded categoricals without specifying cat_features — CatBoost will treat them as numeric and miss the categorical signal entirely. Always identify your categorical columns explicitly. Second, forgetting to fill NaN in categorical columns — numeric NaN is handled natively but categorical NaN causes an error unless you fill it with a placeholder string. Third, using column names in cat_features when training with a scikit-learn Pipeline — inside a Pipeline, column names are not preserved after transformations, so use column indices instead. Fourth, setting iterations too low and not using early stopping — CatBoost with early stopping and a high iteration ceiling almost always beats a manually tuned fixed number of trees, because the optimal stopping point varies by dataset.

When to Choose CatBoost Over XGBoost or LightGBM

The answer depends on your data. If your dataset has significant categorical columns — more than two or three, or any with high cardinality — CatBoost is the most natural choice. The preprocessing you would need to do for XGBoost (one-hot encoding, target encoding with proper cross-validation to avoid leakage) is handled correctly and automatically inside CatBoost. This is not a minor convenience: improper categorical encoding is one of the most common sources of subtle data leakage in ML pipelines, and CatBoost eliminates the entire problem category. If your dataset is predominantly numeric and large (millions of rows), LightGBM’s leaf-wise tree growth is typically faster to train than CatBoost’s symmetric trees. For medium-sized datasets where training speed is not the bottleneck, CatBoost’s ordered boosting often produces better generalisation than XGBoost or LightGBM with equivalent hyperparameter tuning effort. Benchmark all three on your specific data — the answer varies by dataset, and all three are fast enough to trial.

CatBoost in Production

CatBoost models save to a compact binary format (.cbm) that loads quickly and includes all preprocessing metadata — no separate encoders or scalers to manage. The library ships with a C API and supports ONNX export for deployment outside Python environments. For Python serving, the model loads in under a second even for large models and predict() is fast enough for real-time use with individual rows. CatBoost also provides a built-in model analysis tool (catboost.app) for visual inspection of feature importance and prediction explanations via SHAP values, which is useful for stakeholder reporting on production models. The combination of a well-designed Python API, compact model format, and built-in explainability makes CatBoost a solid production choice, not just a training-time convenience. The .cbm format is versioned and backward-compatible, so models trained today will still load correctly after library upgrades — a practical reliability guarantee that matters in long-lived production systems. And for teams that need to serve predictions without a Python runtime, the ONNX export path keeps CatBoost viable across any inference infrastructure.

Leave a Comment