https://optuna.org/

Optuna Logo

  * Key Features
  * Code Examples
  * Installation
  * Dashboard
  * Blog
  * Videos
  * Paper
  * Community

Optimize Your Optimization

An open source hyperparameter optimization framework to automate
hyperparameter search

Key Features

Eager search spaces

Automated search for optimal hyperparameters using Python
conditionals, loops, and syntax

State-of-the-art algorithms

Efficiently search large spaces and prune unpromising trials for
faster results

Easy parallelization

[gif]

Parallelize hyperparameter searches over multiple threads or
processes without modifying code

  * Github
  * Twitter
  * LinkedIn
  * Tutorials
  * Docs

Code Examples

Optuna is framework agnostic. You can use it with any machine
learning or deep learning framework.

  * [gif] Quick Start
  *  
  * [gif] Chainer
  * [gif] TensorFlow
  * Keras
  * [gif] MXNet
  * [gif] Scikit-Learn
  * XGBoost
  * LightGBM
  * Other

A simple optimization problem:

 1. Define objective function to be optimized. Let's minimize (x - 2)
    ^2
 2. Suggest hyperparameter values using trial object. Here, a float
    value of x is suggested from -10 to 10
 3. Create a study object and invoke the optimize method over 100
    trials

import optuna

def objective(trial):
    x = trial.suggest_float('x', -10, 10)
    return (x - 2) ** 2

study = optuna.create_study()
study.optimize(objective, n_trials=100)

study.best_params  # E.g. {'x': 2.002108042}

colab.research.google Open in Colab Open in Colab

You can optimize PyTorch hyperparameters, such as the number of
layers and the number of hidden nodes in each layer, in three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import torch

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):

    # 2. Suggest values of the hyperparameters using a trial object.
    n_layers = trial.suggest_int('n_layers', 1, 3)
    layers = []

    in_features = 28 * 28
    for i in range(n_layers):
        out_features = trial.suggest_int(f'n_units_l{i}', 4, 128)
        layers.append(torch.nn.Linear(in_features, out_features))
        layers.append(torch.nn.ReLU())
        in_features = out_features
    layers.append(torch.nn.Linear(in_features, 10))
    layers.append(torch.nn.LogSoftmax(dim=1))
    model = torch.nn.Sequential(*layers).to(torch.device('cpu'))
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

You can optimize Chainer hyperparameters, such as the number of
layers and the number of hidden nodes in each layer, in three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import chainer

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):

    # 2. Suggest values of the hyperparameters using a trial object.
    n_layers = trial.suggest_int('n_layers', 1, 3)
    layers = []

    for i in range(n_layers):
        n_units = trial.suggest_int(f'n_units_l{i}', 4, 128, log=True)
        layers.append(L.Linear(None, n_units))
        layers.append(F.relu)
    layers.append(L.Linear(None, 10))

    model = L.Classifier(chainer.Sequential(*layers))
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

You can optimize TensorFlow hyperparameters, such as the number of
layers and the number of hidden nodes in each layer, in three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import tensorflow as tf

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):

    # 2. Suggest values of the hyperparameters using a trial object.
    n_layers = trial.suggest_int('n_layers', 1, 3)
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Flatten())
    for i in range(n_layers):
        num_hidden = trial.suggest_int(f'n_units_l{i}', 4, 128, log=True)
        model.add(tf.keras.layers.Dense(num_hidden, activation='relu'))
    model.add(tf.keras.layers.Dense(CLASSES))
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

You can optimize Keras hyperparameters, such as the number of filters
and kernel size, in three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import keras

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):
    model = Sequential()

    # 2. Suggest values of the hyperparameters using a trial object.
    model.add(
        Conv2D(filters=trial.suggest_categorical('filters', [32, 64]),
               kernel_size=trial.suggest_categorical('kernel_size', [3, 5]),
               strides=trial.suggest_categorical('strides', [1, 2]),
               activation=trial.suggest_categorical('activation', ['relu', 'linear']),
               input_shape=input_shape))
    model.add(Flatten())
    model.add(Dense(CLASSES, activation='softmax'))

    # We compile our model with a sampled learning rate.
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
    model.compile(loss='sparse_categorical_crossentropy', optimizer=RMSprop(lr=lr), metrics=['accuracy'])
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

You can optimize MXNet hyperparameters, such as the number of layers
and the number of hidden nodes in each layer, in three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import mxnet as mx

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):

    # 2. Suggest values of the hyperparameters using a trial object.
    n_layers = trial.suggest_int('n_layers', 1, 3)

    data = mx.symbol.Variable('data')
    data = mx.sym.flatten(data=data)
    for i in range(n_layers):
        num_hidden = trial.suggest_int(f'n_units_l{i}', 4, 128, log=True)
        data = mx.symbol.FullyConnected(data=data, num_hidden=num_hidden)
        data = mx.symbol.Activation(data=data, act_type="relu")

    data = mx.symbol.FullyConnected(data=data, num_hidden=10)
    mlp = mx.symbol.SoftmaxOutput(data=data, name="softmax")
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

You can optimize Scikit-Learn hyperparameters, such as the C
parameter of SVC and the max_depth of the RandomForestClassifier, in
three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import sklearn

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):

    # 2. Suggest values for the hyperparameters using a trial object.
    classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
    if classifier_name == 'SVC':
         svc_c = trial.suggest_float('svc_c', 1e-10, 1e10, log=True)
         classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32, log=True)
        classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth, n_estimators=10)
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

You can optimize XGBoost hyperparameters, such as the booster type
and alpha, in three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import xgboost as xgb

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):
    ...

    # 2. Suggest values of the hyperparameters using a trial object.
    param = {
        'silent': 1,
        'objective': 'binary:logistic',
        'booster': trial.suggest_categorical('booster', ['gbtree', 'gblinear', 'dart']),
        'lambda': trial.suggest_float('lambda', 1e-8, 1.0, log=True),
        'alpha': trial.suggest_float('alpha', 1e-8, 1.0, log=True)
    }

    bst = xgb.train(param, dtrain)
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

You can optimize LightGBM hyperparameters, such as boosting type and
the number of leaves, in three steps:

 1. Wrap model training with an objective function and return
    accuracy
 2. Suggest hyperparameters using a trial object
 3. Create a study object and execute the optimization

import lightgbm as lgb

import optuna

# 1. Define an objective function to be maximized.
def objective(trial):
    ...

    # 2. Suggest values of the hyperparameters using a trial object.
    param = {
        'objective': 'binary',
        'metric': 'binary_logloss',
        'verbosity': -1,
        'boosting_type': 'gbdt',
        'lambda_l1': trial.suggest_float('lambda_l1', 1e-8, 10.0, log=True),
        'lambda_l2': trial.suggest_float('lambda_l2', 1e-8, 10.0, log=True),
        'num_leaves': trial.suggest_int('num_leaves', 2, 256),
        'feature_fraction': trial.suggest_float('feature_fraction', 0.4, 1.0),
        'bagging_fraction': trial.suggest_float('bagging_fraction', 0.4, 1.0),
        'bagging_freq': trial.suggest_int('bagging_freq', 1, 7),
        'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
    }

    gbm = lgb.train(param, dtrain)
    ...
    return accuracy

# 3. Create a study object and optimize the objective function.
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

See full example on Github

Check more examples including PyTorch Ignite, Dask-ML and MLFlow at
our Github repository.
It also provides the visualization demo as follows:

from optuna.visualization import plot_intermediate_values

...
plot_intermediate_values(study)

[gif]
See full example on Github

Installation

Optuna can be installed with pip. Python 3.7 or newer is supported.

% pip install optuna

Details

  * Github
  * Twitter
  * Tutorials
  * Docs

Dashboard

Optuna Dashboard

Optuna Dashboard is a real-time web dashboard for Optuna. You can
check the optimization history, hyperparameter importances, etc. in
graphs and tables.

% pip install optuna-dashboard
% optuna-dashboard sqlite:///db.sqlite3

Optuna Dashboard is also available as extensions for Jupyter Lab and
Visual Studio Code.

VS Code Extension

VS Code Extension

To use, install the extension, right-click the SQLite3 files in the
file explorer and select the "Open in Optuna Dashboard" from the
dropdown menu.

Jupyter Lab Extension

Jupyter Lab Extension

% pip install jupyterlab jupyterlab-optuna

GitHub Repository Documentation VS Code Extension (Marketplace)

Blog

 
Mar 26, 2024

Prototyping a Faster Optuna Implementation in Rust

Announcing Optuna 3.4 We've started prototyping a faster Optuna
implementation in Rust, featuring Python, JavaScript, and C bindings.
Masashi Shibata explains the motivation and the potential benefits
the Rust-based Optuna implementation could provide.
 
Mar 18, 2024

Announcing Optuna 3.6

Announcing Optuna 3.6 We have released v3.6, the latest version of
the black-box optimization framework, Optuna. This release includes
various new features, refactoring, and bug fixes. In this blog, we
will be covering the highlights of v3.6 and many feature
improvements.
 
Oct 31, 2023

The Internal of Optuna's Jupyter Lab and VS Code Extensions

Announcing Optuna 3.4 As we announced in the Optuna 3.4 release blog,
we have introduced Optuna Jupyter Lab extension and the VS Code
extension. These extensions enable easy access to the Optuna
Dashboard within Jupyter Lab and VS Code, making it convenient for
you to view your optimization history, hyperparameter importances,
etc. in graphs and tables.
 
Oct 17, 2023

Announcing Optuna 3.4

Announcing Optuna 3.4 We are pleased to announce the release of
Optuna 3.4! Optuna 3.4 newly supports preferential optimization, an
Optuna artifact, a Jupyter Lab extension, and a VS Code extension.
This blog will walk you through these updates and explain how to use
new features. Let's take a closer look at these new and improved
features.
See more stories on Medium

Videos

Paper

If you use Optuna in a scientific publication, please use the
following citation:

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019.
Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD.

View Paper arXiv Preprint

Bibtex entry:

@inproceedings{optuna_2019,
    title={Optuna: A Next-generation Hyperparameter Optimization Framework},
    author={Akiba, Takuya and Sano, Shotaro and Yanase, Toshihiko and Ohta, Takeru and Koyama, Masanori},
    booktitle={Proceedings of the 25th {ACM} {SIGKDD} International Conference on Knowledge Discovery and Data Mining},
    year={2019}
}

Community

Zhong Wen Wen Dang 

  * Github
  * Twitter
  * Tutorials
  * Docs

  * Preferred Networks, Inc.
  * Brand Guidelines

(c) 2017- Preferred Networks, Inc. All rights reserved.
TensorFlow, the TensorFlow logo and any related marks are trademarks
of Google Inc.
Scikit-learn is distributed under the new BSD license.
Apache MXNet and the MXNet logo are either registered trademarks or
trademarks of
the Apache Software Foundation in the United States and/or other
countries.
| Design: (c) 2017 Creative Tim