A Coding Guide to Implement Advanced Hyperparameter Optimization with Optuna

Advanced Hyperparameter Optimization with Optuna

Optuna is a powerful hyperparameter optimization framework enabling systematic exploration of pruning, multi-objective optimization, custom callbacks, and rich visualization. This tutorial demonstrates how to shape search spaces, accelerate experiments, and extract insights for model improvement using real datasets and efficient search strategies.

The ideal model training process often clashes with the realities of computational cost and time constraints; exhaustive searches are impractical. Without efficient optimization, projects can stall or deliver suboptimal results, potentially costing significant resources.

Key Insights

• Pruning efficiency: 36% of trials were pruned in Study 1, accelerating the optimization process.
• Pareto optimization: Optuna automatically constructs a Pareto front to compare trade-offs between competing objectives, such as accuracy and model complexity.
• Custom callbacks: Early stopping callbacks can significantly reduce training time by halting trials that show no improvement after a defined number of iterations, as demonstrated with a 15-round early stopping callback in Study 3.

Working Example

import optuna
from optuna.pruners import MedianPruner
from optuna.samplers import TPESampler
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, KFold
from sklearn.ensemble import GradientBoostingClassifier

def objective_with_pruning(trial):
    X, y = load_breast_cancer(return_X_y=True)
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 50, 200),
        'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),
        'subsample': trial.suggest_float('subsample', 0.6, 1.0),
        'max_features': trial.suggest_categorical('max_features', ['sqrt', 'log2', None]),
    }
    model = GradientBoostingClassifier(**params, random_state=42)
    kf = KFold(n_splits=3, shuffle=True, random_state=42)
    scores = []
    for fold, (train_idx, val_idx) in enumerate(kf.split(X)):
        X_train, X_val = X[train_idx], X[val_idx]
        y_train, y_val = y[train_idx], y[val_idx]
        model.fit(X_train, y_train)
        score = model.score(X_val, y_val)
        scores.append(score)
        trial.report(np.mean(scores), fold)
        if trial.should_prune():
            raise optuna.TrialPruned()
    return np.mean(scores)

study1 = optuna.create_study(
    direction='maximize',
    sampler=TPESampler(seed=42),
    pruner=MedianPruner(n_startup_trials=5, n_warmup_steps=1)
)
study1.optimize(objective_with_pruning, n_trials=30, show_progress_bar=True)
print(study1.best_value, study1.best_params)

Practical Applications

• Recommendation Systems (Netflix, Spotify): Optimizing model hyperparameters for personalized recommendations to improve user engagement and retention.
• Pitfall: Over-reliance on default hyperparameter settings, leading to suboptimal model performance and missed opportunities for improvement.

References:

• https://www.marktechpost.com/2025/11/17/a-coding-guide-to-implement-advanced-hyperparameter-optimization-with-optuna-using-pruning-multi-objective-search-early-stopping-and-deep-visual-analysis/