A Practical HPO Cookbook

Abstract

On the one hand, hyperparameter optimization (HPO) can be crucial to leverage the full potential of machine learning models and pipelines, even in the age of agentic workflows and foundation models. On the other hand, HPO can be quite frustrating for new users because the tools suggest very simple use, but in practice, we need to follow sound workflows to ensure reliable results that actually pay off and do not waste compute or inflate expectations. This is a practical step-by-step guide to HPO with no equations or definitions.