A Practical HPO Cookbook

Abstract

On the one hand, hyperparameter optimization can be crucial to leverage the full potential of machine learning models and pipelines, even in the age of agentic workflows and foundation models. On the other hand, hyperparameter optimization can be quite frustrating for new users because tools suggest very simple use, but in practice, we need to pay attention to sound workflow to ensure reliable results that actually pay off and do not waste compute or inflate expectations. This is a practical step-by-step guide to hyperparameter optimization with no equations or definitions.