This post has been a long time coming. Realistically it should have been as part of my Machine Learning with Zeek series. Guess I got sloppy and not included it or was just too lazy. Whatever the reason it is here now.
Another more important reason, is when teaching the SANS SEC595: Machine Learning and AI for Cyber Security Professionals course, the question always arise - before we get to that section - but how do I know how many neurons to use, which activation function, etc.? This post is meant to provide some additional insights with a lot more details than I provide when doing my demos. Make no mistake, my demos make the point. However, I thought I can be a bit more involved here.
Now to be specific, when we talk about hyperparameter optimization in class, we talk about Grid Search, Random Search Bayesian, early stopping as well as genetic optimization. Do note, there are many strategies to address this optimization problem. In this post however, I will focus specifically on Hyperband, which is another strategy for hyperparameter optimization
Why do I need to worry about this?
Simple! In machine learning, it is not only about finding the right algorithm but more importantly finding the right combination of hyperparameters. Or put another way for neural networks, finding the right values for the weights (w) and the biases (b). Traditional machine learning, one my say, finding the right values for the slope (m) and the bias/intercept (b).
What is Hyperband?
Hyperband was introduced in the paper, "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization" by Li, et al.
Hyperband focuses on speeding up Random Search. It sees the problem as an adaptive resource allocation problem and allocates resources among randomly sampled hyperparameter configurations. It also leverages early-stopping strategy, which means if the model is not improving after a certain number of epochs, then training should be stopped for that model.
Along with early stopping, one can train on a subset of data or even a subset of features.
One of the key strategies used by Hyperband is Successive Halving:
The Successive Halving algorithm is extended by hyperband and called as a subroutine. The core idea behind the original Successive Halving is that at the first iteration, a small amount of resources - the number of samples, etc. - along with all hyperparameters candidates are used.
As we move to the second iteration, we get the best half of the performing hyperparameters - meaning we discard the set that does not work well - and double the resources - once again the number of samples.
The process above gets repeated until the best combination of hyperparameters are found.
This strategy is also called a bandit approach.
Remember, Hyperband's aim is to extend Successive Halving. One of the ways it does this is by randomly distributing resources to each batch rather than uniformly to explore different convergence behaviours.
To ensure everyone can follow along, we will use a toy dataset to build our intuition. We also look at this from the perspective of both Tensorflow and PyTorch.
See the Tensorflow / KerasTuner Notebook and
The PyTorch / Ray Tune
In the end, when we look at the test accuracy, we see they are about the same.
References:
https://arxiv.org/pdf/1603.06560
https://scikit-learn.org/stable/auto_examples/model_selection/plot_successive_halving_iterations.html
https://www.youtube.com/watch?v=eqokKei1aEo&t=221s
https://www.youtube.com/watch?v=U5wbvPzW57M
https://www.youtube.com/watch?v=JUrU92fTxpo
Coding supported by www.duckai.com
