Hyperparameters Optimization of Machine Learning Algorithms

Abstract

Machine learning algorithms have been widely used in various applications and areas. Hyper-parameters of machine learning models must be optimized in order to fit them into various problems. Performance of machine learning models is highly dependent on the selection of the best hyper parameter configuration. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this research, we have provided a comprehensive analysis of several state-of-the-art hyper-parameter optimization techniques including their strengths and limitations, and discussed how to apply them to various machine learning and deep learning models. Also, on the basis of extensive literature review, a taxonomy for the hyper-parameters optimization techniques is proposed. Moreover, experiment is conducted on benchmark dataset to compare the performance of different optimization methods. This research will help industrial users, data analysts, and researchers to better develop machine learning or deep learning models by identifying the proper hyper-parameter configurations effectively.