Machine learning experiments can quickly devolve into chaos without proper structure. You run dozens of experiments, tweak parameters, try different features, and suddenly you can’t remember which combination produced your best results. Your notebook becomes a graveyard of commented-out code, random cells executed out of order, and results you can’t reproduce. Sound familiar?
Kaggle notebooks offer a perfect environment for ML experimentation, but their flexibility becomes a liability without disciplined practices. Top Kaggle competitors don’t just write better code—they structure their experiments systematically, making every iteration trackable and every result reproducible. This guide reveals the battle-tested practices that separate chaotic experimentation from professional ML workflow, transforming your notebooks from messy scripts into reliable research tools.
Structuring Your Notebook for Reproducibility
Reproducibility is the foundation of scientific ML work. If you can’t reproduce your results, you can’t learn from them, can’t debug them, and certainly can’t trust them. Every Kaggle notebook should be structured so that anyone—including future you—can run it top to bottom and get identical results.
Start every notebook with a configuration cell that sets all random seeds and defines key parameters. This single cell controls randomness across all libraries:
import numpy as np
import pandas as pd
import random
import os
from sklearn.model_selection import train_test_split
# Set all random seeds for reproducibility
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
os.environ['PYTHONHASHSEED'] = str(SEED)
# TensorFlow/Keras seeds (if using deep learning)
try:
import tensorflow as tf
tf.random.set_seed(SEED)
except ImportError:
pass
# PyTorch seeds (if using PyTorch)
try:
import torch
torch.manual_seed(SEED)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(SEED)
except ImportError:
pass
# Configuration parameters
CONFIG = {
'seed': SEED,
'test_size': 0.2,
'n_folds': 5,
'verbose': True,
'save_predictions': True
}
print(f"Random seed set to {SEED}")
print(f"Configuration: {CONFIG}")
Organize your notebook into clearly defined sections using markdown headers. Follow this proven structure:
1. Setup and Configuration – Import libraries, set seeds, define parameters 2. Data Loading and Exploration – Load data, initial inspection, basic statistics
3. Data Preprocessing – Cleaning, handling missing values, type conversions 4. Feature Engineering – Create new features, transformations, encoding 5. Model Training – Build and train models with cross-validation 6. Evaluation and Analysis – Metrics, visualizations, error analysis 7. Predictions and Submission – Generate predictions for test set
This structure makes your notebook easy to navigate and ensures logical flow. Each section builds on previous ones without circular dependencies. Anyone can understand your approach by reading the section headers alone.
Version control your hyperparameters by keeping them in the CONFIG dictionary rather than hardcoding them throughout your notebook. This makes experimentation easier—change one value in CONFIG and rerun, rather than hunting through cells to update scattered magic numbers. Document what each parameter does with comments.
Managing Experiments and Tracking Results
The difference between random experimentation and systematic improvement is tracking. Without proper tracking, you waste time repeating failed experiments, forget what worked, and can’t identify which changes actually improved performance.
Create an experiment log directly in your notebook. Use a pandas DataFrame to track every experiment with its parameters and results:
# Initialize experiment tracking
experiment_log = pd.DataFrame(columns=[
'experiment_id', 'date', 'model', 'features',
'cv_score', 'cv_std', 'best_params', 'notes'
])
def log_experiment(model_name, features_used, cv_score, cv_std, params, notes=''):
"""Log experiment results to tracking dataframe"""
global experiment_log
experiment_id = len(experiment_log) + 1
timestamp = pd.Timestamp.now().strftime('%Y-%m-%d %H:%M')
new_row = pd.DataFrame([{
'experiment_id': experiment_id,
'date': timestamp,
'model': model_name,
'features': str(features_used),
'cv_score': cv_score,
'cv_std': cv_std,
'best_params': str(params),
'notes': notes
}])
experiment_log = pd.concat([experiment_log, new_row], ignore_index=True)
print(f"\n{'='*60}")
print(f"Experiment #{experiment_id} logged")
print(f"Model: {model_name}")
print(f"CV Score: {cv_score:.4f} (+/- {cv_std:.4f})")
print(f"{'='*60}\n")
return experiment_id
# Example usage after running an experiment
# exp_id = log_experiment('RandomForest', selected_features, 0.8234, 0.0156,
# {'n_estimators': 200}, 'Added interaction features')
This tracking system creates a permanent record of every experiment. You can sort by score to find your best models, compare similar experiments to understand what changed, and identify patterns in what works. Save this log periodically so you never lose your experimental history.
Use meaningful variable names that indicate what each object contains. Instead of df1, df2, df3, use train_raw, train_processed, train_features. Instead of model1, use rf_baseline or xgb_tuned. Clear names make your notebook self-documenting and prevent errors from using the wrong variable.
Comment your code strategically. Don’t comment obvious operations like # Load data before pd.read_csv(). Instead, explain why you’re doing something or document assumptions: # Use median instead of mean because of extreme outliers or # Remove features with >50% missing values based on EDA findings. Your comments should answer “why” questions that aren’t obvious from the code itself.
Implementing Robust Cross-Validation Strategies
Cross-validation is non-negotiable for reliable ML experiments. Single train-test splits give misleading results because performance varies based on which data points end up in which set. Proper cross-validation reveals true model performance and prevents overfitting to validation sets.
Implement stratified k-fold cross-validation for classification problems to ensure each fold maintains the same class distribution as the full dataset. This is especially critical for imbalanced datasets where random splits might create folds with missing classes:
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
import warnings
warnings.filterwarnings('ignore')
def robust_cross_validation(model, X, y, n_folds=5, metric='roc_auc'):
"""
Perform stratified k-fold cross-validation with detailed reporting
Parameters:
- model: sklearn-compatible model
- X: feature matrix
- y: target variable
- n_folds: number of CV folds
- metric: scoring metric
Returns:
- Dictionary with scores and statistics
"""
skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=CONFIG['seed'])
scores = cross_val_score(model, X, y, cv=skf, scoring=metric, n_jobs=-1)
results = {
'scores': scores,
'mean': scores.mean(),
'std': scores.std(),
'min': scores.min(),
'max': scores.max(),
'cv_range': scores.max() - scores.min()
}
# Print detailed results
print(f"\n{metric.upper()} Cross-Validation Results:")
print(f"{'='*50}")
for i, score in enumerate(scores, 1):
print(f"Fold {i}: {score:.4f}")
print(f"{'='*50}")
print(f"Mean: {results['mean']:.4f}")
print(f"Std: {results['std']:.4f}")
print(f"Range: {results['cv_range']:.4f}")
print(f"{'='*50}\n")
# Warning for high variance
if results['std'] > 0.05:
print("⚠️ WARNING: High variance across folds - model may be unstable")
return results
# Example usage
# model = RandomForestClassifier(n_estimators=100, random_state=CONFIG['seed'])
# cv_results = robust_cross_validation(model, X_train, y_train, n_folds=5)
Pay attention to cross-validation variance. High variance (large standard deviation) indicates your model performs inconsistently across different data splits. This suggests either the model is overfitting, your features are unreliable, or your dataset has distinct subgroups that should be handled separately. Low variance with good mean scores indicates a robust model.
For time-series data, never use standard k-fold cross-validation. Time-series requires time-based splits where you always train on past data and validate on future data. Use TimeSeriesSplit from sklearn or create custom splits that respect temporal order:
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for fold, (train_idx, val_idx) in enumerate(tscv.split(X)):
X_train_fold = X.iloc[train_idx]
X_val_fold = X.iloc[val_idx]
# Train and evaluate
Group-based cross-validation prevents data leakage when you have related records. If predicting whether customers will churn and the same customer appears multiple times, use GroupKFold to ensure all instances of a customer stay together in either train or validation:
from sklearn.model_selection import GroupKFold
gkf = GroupKFold(n_splits=5)
for train_idx, val_idx in gkf.split(X, y, groups=df['customer_id']):
# All records for a customer stay together
pass
Optimizing Code Performance and Memory Usage
Kaggle notebooks have resource limits—9 hours of CPU time or 12 hours of GPU time per session, with 16GB RAM for CPU notebooks and 13GB for GPU notebooks. Efficient code lets you run more experiments within these constraints. Poor code wastes time and hits memory limits, killing your kernel and losing work.
Reduce memory usage by selecting appropriate data types. Pandas defaults to int64 and float64, but most datasets don’t need 64-bit precision. Downcasting to smaller types can cut memory usage by 50-75%:
def reduce_memory_usage(df, verbose=True):
"""Reduce dataframe memory usage by downcasting numeric types"""
start_mem = df.memory_usage().sum() / 1024**2
for col in df.columns:
col_type = df[col].dtype
if col_type != object:
c_min = df[col].min()
c_max = df[col].max()
if str(col_type)[:3] == 'int':
if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
df[col] = df[col].astype(np.int8)
elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
df[col] = df[col].astype(np.int16)
elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
df[col] = df[col].astype(np.int32)
else:
if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
df[col] = df[col].astype(np.float32)
end_mem = df.memory_usage().sum() / 1024**2
reduction = 100 * (start_mem - end_mem) / start_mem
if verbose:
print(f'Memory usage decreased from {start_mem:.2f}MB to {end_mem:.2f}MB')
print(f'Memory reduced by {reduction:.1f}%')
return df
# Usage
# train_df = reduce_memory_usage(train_df)
Delete large objects you no longer need and explicitly call garbage collection. After feature engineering, if you’re keeping both original and processed dataframes, delete the original if you won’t need it again:
import gc
# Delete unnecessary objects
del train_raw, large_intermediate_df
gc.collect() # Force garbage collection to free memory immediately
Use generators and chunking for very large datasets. Instead of loading everything into memory, process data in chunks:
# Process large file in chunks
chunk_size = 100000
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
# Process each chunk
processed_chunk = preprocess(chunk)
# Aggregate results
Vectorize operations instead of using loops. Pandas and NumPy operations are orders of magnitude faster than Python loops:
# Slow - iterates row by row
for i in range(len(df)):
df.loc[i, 'total'] = df.loc[i, 'price'] * df.loc[i, 'quantity']
# Fast - vectorized operation
df['total'] = df['price'] * df['quantity']
# Also fast - using apply with axis=1 when vectorization isn't possible
df['complex_feature'] = df.apply(lambda row: complex_calculation(row), axis=1)
Cache expensive computations using Kaggle’s dataset feature. If preprocessing takes hours, save the processed data as a dataset and reload it in future notebooks:
# After expensive preprocessing
processed_train.to_csv('processed_train.csv', index=False)
processed_test.to_csv('processed_test.csv', index=False)
# In future notebooks, load processed data directly
# (after uploading as Kaggle dataset)
Creating Reusable Functions and Modular Code
Code reuse accelerates experimentation. When you need to preprocess data the same way multiple times, test different models with the same pipeline, or apply the same transformations to train and test sets, functions ensure consistency and save time.
Wrap your preprocessing pipeline in functions that can be applied identically to any dataset:
def preprocess_data(df, is_train=True):
"""
Standard preprocessing pipeline applicable to train and test data
Parameters:
- df: input dataframe
- is_train: whether this is training data (affects target handling)
Returns:
- Preprocessed dataframe
"""
df = df.copy() # Avoid modifying original
# Handle missing values
numeric_cols = df.select_dtypes(include=[np.number]).columns
df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].median())
categorical_cols = df.select_dtypes(include=['object']).columns
df[categorical_cols] = df[categorical_cols].fillna('Unknown')
# Encode categorical variables
for col in categorical_cols:
if col != 'target': # Don't encode target variable
df[col] = pd.Categorical(df[col]).codes
# Feature engineering
if 'age' in df.columns and 'income' in df.columns:
df['age_income_ratio'] = df['age'] / (df['income'] + 1)
return df
# Apply to both train and test consistently
train_processed = preprocess_data(train_df, is_train=True)
test_processed = preprocess_data(test_df, is_train=False)
Create model training functions that standardize your workflow and make comparing different models trivial:
def train_and_evaluate_model(model, X_train, y_train, model_name='Model'):
"""
Train model with cross-validation and log results
Parameters:
- model: sklearn-compatible model
- X_train: training features
- y_train: training target
- model_name: descriptive name for logging
Returns:
- Trained model, CV scores, and experiment ID
"""
print(f"\nTraining {model_name}...")
# Cross-validation
cv_results = robust_cross_validation(model, X_train, y_train)
# Train on full training set
model.fit(X_train, y_train)
# Log experiment
exp_id = log_experiment(
model_name=model_name,
features_used=list(X_train.columns),
cv_score=cv_results['mean'],
cv_std=cv_results['std'],
params=model.get_params()
)
return model, cv_results, exp_id
# Now comparing models is simple
# rf_model, rf_scores, rf_id = train_and_evaluate_model(
# RandomForestClassifier(n_estimators=100), X, y, 'Random Forest'
# )
# xgb_model, xgb_scores, xgb_id = train_and_evaluate_model(
# XGBClassifier(n_estimators=100), X, y, 'XGBoost'
# )
Build feature engineering functions that can be easily toggled on or off:
def create_time_features(df, datetime_col='timestamp'):
"""Extract temporal features from datetime column"""
df = df.copy()
df[datetime_col] = pd.to_datetime(df[datetime_col])
df['hour'] = df[datetime_col].dt.hour
df['day'] = df[datetime_col].dt.day
df['month'] = df[datetime_col].dt.month
df['dayofweek'] = df[datetime_col].dt.dayofweek
df['is_weekend'] = (df['dayofweek'] >= 5).astype(int)
return df
def create_aggregation_features(df, group_col, agg_col):
"""Create group-based aggregation features"""
df = df.copy()
group_stats = df.groupby(group_col)[agg_col].agg(['mean', 'std', 'min', 'max'])
group_stats.columns = [f'{group_col}_{agg_col}_{stat}' for stat in ['mean', 'std', 'min', 'max']]
df = df.merge(group_stats, on=group_col, how='left')
return df
Documenting Insights and Visualizing Results
Documentation transforms notebooks from code dumps into knowledge repositories. Your notebook should tell a story—what you tried, what you learned, and what worked. Future you and your collaborators need to understand not just what you did, but why.
Use markdown extensively. Before each major section, write a paragraph explaining what you’re about to do and why. After each experiment, write a few sentences about the results and insights. These narrative sections make your notebook readable and searchable.
Create summary visualizations that communicate key findings at a glance:
import matplotlib.pyplot as plt
import seaborn as sns
def plot_cv_comparison(experiment_log, top_n=10):
"""Visualize and compare cross-validation scores across experiments"""
# Sort by CV score and get top N
top_experiments = experiment_log.nlargest(top_n, 'cv_score')
fig, ax = plt.subplots(figsize=(12, 6))
# Bar plot with error bars
x_pos = range(len(top_experiments))
ax.barh(x_pos, top_experiments['cv_score'],
xerr=top_experiments['cv_std'], capsize=5, alpha=0.7)
ax.set_yticks(x_pos)
ax.set_yticklabels([f"Exp {id}: {model}"
for id, model in zip(top_experiments['experiment_id'],
top_experiments['model'])])
ax.set_xlabel('Cross-Validation Score')
ax.set_title(f'Top {top_n} Experiments by CV Score')
ax.grid(axis='x', alpha=0.3)
plt.tight_layout()
plt.show()
# Usage
# plot_cv_comparison(experiment_log, top_n=10)
Visualize feature importance to understand what drives your model:
def plot_feature_importance(model, feature_names, top_n=20):
"""Plot feature importance for tree-based models"""
# Get importance scores
if hasattr(model, 'feature_importances_'):
importance = model.feature_importances_
else:
print("Model doesn't have feature_importances_ attribute")
return
# Create dataframe and sort
importance_df = pd.DataFrame({
'feature': feature_names,
'importance': importance
}).sort_values('importance', ascending=False).head(top_n)
# Plot
plt.figure(figsize=(10, 8))
plt.barh(range(len(importance_df)), importance_df['importance'])
plt.yticks(range(len(importance_df)), importance_df['feature'])
plt.xlabel('Importance')
plt.title(f'Top {top_n} Feature Importance')
plt.gca().invert_yaxis()
plt.tight_layout()
plt.show()
return importance_df
Create a “Key Findings” section at the top of your notebook after you’ve run multiple experiments. Use bullet points to summarize what worked, what didn’t, and actionable insights:
Key Findings:
- Random Forest with 200 trees achieved best CV score (0.8234 ± 0.0156)
- Time-based features (hour, day of week) were top 3 most important features
- Target encoding categorical variables improved performance by 2.3%
- Model shows high variance on Fold 3 – investigate data distribution differences
- Adding polynomial features degraded performance – likely overfitting
Building Submission-Ready Notebooks
Competition notebooks require extra attention to submission formatting and reproducibility. Your notebook must generate predictions in the exact format required by the competition and run successfully when Kaggle’s systems execute it automatically.
Verify output format matches submission requirements exactly. Check column names, data types, and ordering:
# Load sample submission to get correct format
sample_submission = pd.read_csv('/kaggle/input/competition-name/sample_submission.csv')
# Create predictions
predictions = model.predict(X_test)
# Create submission dataframe matching sample format
submission = pd.DataFrame({
'id': test_ids,
'target': predictions
})
# Verify format matches
assert submission.shape == sample_submission.shape, "Shape mismatch"
assert list(submission.columns) == list(sample_submission.columns), "Column mismatch"
# Save submission
submission.to_csv('submission.csv', index=False)
print(f"Submission file created: {submission.shape}")
print(submission.head())
Test notebook reproducibility by using “Run All” frequently during development. Notebooks that work when cells are run interactively often fail when executed sequentially. Issues include: cells that depend on variables defined later, cells that must be run multiple times, or cells that modify global state unpredictably.
Add assertions to catch errors early:
# After data loading
assert len(train_df) > 0, "Training data is empty"
assert 'target' in train_df.columns, "Target column missing"
# After preprocessing
assert train_processed.isnull().sum().sum() == 0, "Null values remain after preprocessing"
# Before submission
assert len(submission) == len(test_df), "Submission has wrong number of predictions"
Comment out or remove exploratory cells that aren’t needed for final submission. Your submission notebook should be clean and focused—load data, preprocess, train, predict, submit. Move extensive EDA and experimental code to separate notebooks.
Conclusion
Professional ML experimentation on Kaggle requires more than just coding skills—it demands systematic practices that ensure reproducibility, efficient resource usage, and clear documentation. By implementing proper notebook structure, experiment tracking, robust validation, and modular code, you transform chaotic trial-and-error into organized scientific inquiry. These practices compound over time, making each subsequent experiment faster and more effective.
The best Kaggle competitors aren’t necessarily those with the deepest theoretical knowledge or access to the most powerful hardware. They’re the ones who approach experiments systematically, learn from every iteration, and build on previous work rather than starting from scratch each time. Adopt these best practices consistently, and you’ll see improvements not just in your competition scores, but in the speed and quality of your entire machine learning workflow.