https://github.com/aimhubio/aim Skip to content Sign up * Why GitHub? Features - + Code review + Project management + Integrations + Actions + Packages + Security + Team management + Hosting + Mobile + Customer stories - + Security - * Team * Enterprise * Explore + Explore GitHub - Learn & contribute + Topics + Collections + Trending + Learning Lab + Open source guides Connect with others + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing Plans - + Compare plans + Contact Sales + Nonprofit - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} aimhubio / aim * Watch 16 * Star 721 * Fork 44 Aim -- a super-easy way to record, search and compare 1000s of ML training runs aimstack.io Apache-2.0 License 721 stars 44 forks Star Watch * Code * Issues 13 * Pull requests 0 * Discussions * Actions * Projects 0 * Wiki * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights main 8 branches 71 tags Go to file Code Clone HTTPS GitHub CLI [https://github.com/a] Use Git or checkout with SVN using the web URL. [gh repo clone aimhub] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio If nothing happens, download the GitHub extension for Visual Studio and try again. Go back Latest commit @gorarakelyan gorarakelyan Document PL hyperparamaters logging (#306) ... 6b4d9e3 Jan 13, 2021 Document PL hyperparamaters logging (#306) 6b4d9e3 Git stats * 607 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time aim Bump up Aim to v2.1.5 Jan 7, 2021 examples Handle NaN or infinite floats passed to artifacts (#299) Jan 7, 2021 tests Check if run exists before initialization (#287) Dec 2, 2020 .gitignore Add .vim to .gitignore Feb 10, 2020 CHANGELOG.md Bump up Aim to v2.1.5 Jan 7, 2021 CONTRIBUTING.md Add CONTRIBUTING document Feb 8, 2020 LICENSE Update MIT license to Apache 2 (#172) Jul 13, 2020 MANIFEST.in update setup py to build cython extensions Sep 20, 2019 README.md Document PL hyperparamaters logging (#306) Jan 13, 2021 main.py Make aim CLI directly accessible from main.py Feb 9, 2020 pytest.ini Add validation of artifact parameters (#283) Nov 30, 2020 requirements.txt Add SDK close method to close dangling experiments (#247) Nov 5, 2020 setup.cfg Tidy up aim and remove some artifacts (#154) Jun 17, 2020 setup.py Reconstruct run metadata file when running close command (#258) Nov 9, 2020 View code README.md [97081166-8] A super-easy way to record, search and compare 1000s of ML training runs PyPI - Python Version PyPI Package Downloads License --------------------------------------------------------------------- Try out Aim at play.aimstack.io Watch the tutorial video Join the Aim community on Slack [104327872-] Integrate seamlessly with your favorite tools [96861310-f723] [97086626-8b3c] [96859323-6ba9] [96861315-f854] Getting started in three steps 1. Install Aim in your training environment $ pip install aim 2. Integrate Aim with your code Flexible integration for any Python script import aim # Save inputs, hparams or any other `key: value` pairs aim.set_params(hyperparam_dict, name='hparams') # Passing name argument is optional ... for step in range(10): # Log metrics to visualize performance aim.track(metric_value, name='metric_name', epoch=epoch_number) ... See documentation here. PyTorch Lightning integration from aim.pytorch_lightning import AimLogger ... trainer = pl.Trainer(logger=AimLogger(experiment='experiment_name')) ... See documentation here. Keras & tf.keras integrations import aim # Save inputs, hparams or any other `key: value` pairs aim.set_params(param_dict, name='params_name') # Passing name argument is optional ... model.fit(x_train, y_train, epochs=epochs, callbacks=[ aim.keras.AimCallback(aim.Session(experiment='experiment_name')) # Use aim.tensorflow.AimCallback in case of tf.keras aim.tensorflow.AimCallback(aim.Session(experiment='experiment_name')) ]) ... See documentation here. 3. Run the training like you are used to and start Aim UI $ aim up Contents * Aim + Contents + Getting Started In Three Steps + Installation + Concepts + Where is the Data Stored + Python Library o aim.track() o aim.set_params() o aim.Session() + Automatic Tracking o TensorFlow and Keras o PyTorch Lightning + Searching Experiments o Search Examples + Command Line Interface o init o version o experiment o up o down o upgrade o pull + TensorBoard Experiments + Contributor Guide Installation To install Aim, you need to have python3 and pip3 installed in your environment 1. Install Aim python package $ pip install aim In order to start Aim UI you need to have Docker installed. $ aim up Concepts Aim doesn't use too many concepts that needs explaining. Still, here are a few details to avoid confusion: * Experiment: it's a grouping mechanism for training runs. Experiments are a way to organize the training runs in a meaningful manner * Run: it's the single training run researchers start These two are separated just to enable effective organization of the research work. Where is the Data Stored When the AI training code is instrumented with Aim Python Library and ran, aim automatically creates a .aim directory where the project is located. All the metadata tracked during training via the Python Library is stored in .aim. Also see aim init - an optional and alternative way to initialize aim repository. Python Library Use Python Library to instrument your training code to record the experiments. The instrumentation only takes 2 lines: import aim Afterwards, simply use the two following functions to track metrics and any params respectively. ... aim.track(metric_val, name='metric_name', epoch=current_epoch) aim.set_params(hyperparam_dict, name='dict_name') ... track aim.track(value, name='metric_name' [, epoch=epoch] [, **context_args]) [source] Parameters * value - the metric value of type int/float to track/log * name - the name of the metric of type str to track/log (preferred divider: snake_case) * epoch - an optional value of the epoch being tracked * context_args - any set of other parameters passed would be considered as key-value context for metrics Examples aim.track(0.01, name='loss', epoch=43, subset='train', dataset='train_1') aim.track(0.003, name='loss', epoch=43, subset='val', dataset='val_1') Once tracked this way, the following search expressions will be enabled: loss if context.subset in (train, val) # Retrieve all losses in both train and val phase loss if context.subset == train and context.dataset in (train_1) # Retrieve all losses in train phase with given datasets Please note that any key-value could be used to track this way and enhance the context of metrics and enable even more detailed search. Search by context example here: set_params aim.set_params(dict_value, name) [source] Parameters * dict_value - Any dictionary relevant to the training * name - A name for dictionaries Examples # really any dictionary can go here hyperparam_dict = { 'learning_rate': 0.0001, 'batch_siz': 32} aim.set_params(hyperparam_dict, name='params') The following params can be used later to perform the following search experssions loss if params.learning_rate < 0.01 # All the runs where learning rate is less than 0.01 loss if params.learning_rate == 0.0001 and params.batch_size == 32 # all the runs where learning rate is 0.0001 and batch_size is 32 Note: if the set_params is called several times with the same name all the dictionaries will add up in one place on the UI. flush aim.flush() [source] Aim calculates intermediate values of metrics for aggregation during tracking. This method is called at a given frequency(see Session) and at the end of the run automatically. Use this command to flush those values to disk manually. Session Use Session to specify custom .aim directory or the experiment from the code. Class aim.Session()[source] Parameters * repo - Full path to parent directory of Aim repo - the .aim directory. By default current working directory. * experiment - A name of the experiment. By default default. See concepts * flush_frequency - The frequency per step to flush intermediate aggregated values of metrics to disk. By default per 128 step. * block_termination - If set to True process will wait until all tasks are completed, otherwise pending tasks will be killed at process exit. By default True. * run - A name of the run. If run name is not specified, universally unique identifier will be generated. Returns * Session object to attribute recorded training run to. Methods * track() - Tracks metrics within the session * set_params() - Sets session params * flush() - Flushes intermediate aggregated metrics to disk. This method is called at a given frequency and at the end of the run automatically. * close() - Closes the session. If not invoked, the session will be automatically closed when the training is done. Examples * Here are a few examples of how to use the aim.Session in code Automatic Tracking Automatic tracking allows you to track metrics without the need for explicit track statements. TensorFlow and Keras Pass an instance of aim.tensorflow.AimCallback to the trainer callbacks list. Note: Logging for pure keras is handled by aim.keras.AimCallback Parameters * session - Aim Session instance (optional) Examples from aim import Session from aim.tensorflow import AimCallback # Use `from aim.keras import AimCallback` in case of keras ... aim_session = Session(experiment='experiment_name') model.fit(x_train, y_train, epochs=epochs, callbacks=[ AimCallback(aim_session) ]) ... TensorFlow v1 full example here TensorFlow v2 full example here Keras full example here PyTorch Lightning Pass aim.pytorch_lightning.AimLogger instance as logger to pl.Trainer to log metrics and parameters automatically. Parameters * repo - Full path to parent directory of Aim repo - the .aim directory (optional) * experiment - A name of the experiment (optional) * train_metric_prefix - The prefix of metrics names collected in the training loop. By default train_ (optional) * test_metric_prefix - The prefix of metrics names collected in the test loop. By default test_ (optional) * val_metric_prefix - The prefix of metrics names collected in the validation loop. By default val_ (optional) * flush_frequency - The frequency per step to flush intermediate aggregated values of metrics to disk. By default per 128 step. (optional) Examples from aim.pytorch_lightning import AimLogger ... # Initialize Aim PL logger instance aim_logger = AimLogger(experiment='pt_lightning_exp') # Log parameters (optional) aim_logger.log_hyperparams({ "max_epochs": 10, }) trainer = pl.Trainer(logger=aim_logger) trainer.fit(model, train_loader, val_loader) ... Full example here Searching Experiments AimQL is a super simple, python-like search that enables rich search capabilities to search experiments. Here are the ways you can search on Aim: * Search by experiment name - experiment == {name} * Search by run - run.hash == "{run_hash}" or run.hash in (" {run_hash_1}", "{run_hash_2}") or run.archived is True * Search by param - params.{key} == {value} * Search by context - context.{key} == {value} Search Examples * Display the losses and accuracy metrics of experiments whose learning rate is 0.001: + loss, accuracy if params.learning_rate == 0.001 * Display the train loss of experiments whose learning rate is greater than 0.0001: + loss if context.subset == train and params.learning_rate > 0.0001 Check out this demo project deployment to play around with search. Command Line Interface Aim CLI offers a simple interface to easily organize and record your experiments. Paired with the Python Library, Aim is a powerful utility to record, search and compare AI experiments. Here are the set of commands supported: Command Description init Initialize the aim repository. version Displays the version of aim cli currently installed. experiment Creates a new experiment to group similar training runs into. up Runs Aim web UI for the given repo down Turn off the UI upgrade Upgrade the UI to its latest version pull Pull the UI of the given version init This step is optional. Initialize the aim repo to record the experiments. $ aim init Creates .aim directory to save the recorded experiments to. Running aim init in an existing repository will prompt the user for re-initialization. Beware: Re-initialization of the repo clears .aim folder from previously saved data and initializes new repo. Note: This command is not necessary to be able to get started with Aim as aim is automatically initializes with the first aim function call. version Display the Aim version installed. $ aim version experiment Create new experiments to organize the training runs. Here is how it works: $ aim experiment COMMAND [ARGS] Command Args Description add -n | --name Add new experiment with a given name. checkout -n | --name Switch/checkout to an experiment with given name. ls List all the experiments of the repo. rm -n | --name Remove an experiment with the given name. Disclaimer: Removing the experiment also removes the recorded experiment runs data. up Start the Aim web UI locally. Aim UI is a Docker container that mounts the .aim folder and lets researchers manage, search and start new training runs. $ aim up [ARGS] Args Description -h | --host Specify host address. -p | --port Specify port to listen to. -v | --version Version of Aim UI to run. Default latest. --repo Path to parent directory of .aim repo. Current working directory by default -d | --detach Run Aim UI in detached mode. --tf_logs Use Aim to search and compare TensorBoard experiments. More details in TensorBoard Experiments Disclaimer: UI uses docker container to run and having docker installed in the training environment is mandatory for the UI to run. Most of the environments nowadays have docker preinstalled or installed for other purposes so this should not be a huge obstacle to get started with Aim UI. Please make sure to run aim up in the directory where .aim is located. down Turn off Aim UI manually: $ aim down [ARGS] Args Description --repo Path to parent directory of .aim repo. Current working directory by default upgrade Upgrade Aim UI to its latest version: $ aim upgrade pull Pulls Aim UI of the given version: $ aim pull -v TensorBoard Experiments Easily run Aim on experiments visualized by TensorBoard. Here is how: $ aim up --tf_logs path/to/logs This command will spin up Aim on the TensorFlow summary logs and load the logs recursively from the given path. Use tf: prefix to select and display metrics logged with tf.summary in the dashboard, for example tf:accuracy. Tensorboard search example here About Aim -- a super-easy way to record, search and compare 1000s of ML training runs aimstack.io Topics machine-learning training-tracking experiment-tracking mlops pytorch tensorflow keras pytorch-lightning nlp reinforcement-learning Resources Readme License Apache-2.0 License Releases 71 tags Contributors 8 * @gorarakelyan * @SGevorg * @mike1808 * @jialin-wu-02 * @jamesj-jiao * @bhaveshk658 * @bluetyson Languages * Python 100.0% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Help * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.