Addressing the Non-Differentiability in ReLU through a Smooth Polynomial Approximation to Enhance ReLU’s Performance in Image Classification Problems

Abstract

Neural Networks rely heavily on Activation Functions. These functions determine how the network responds to its inputs and how it learns. Traditional activation functions, such as the rectified linear unit (ReLU) and the exponential linear unit (ELU), are effective for many tasks. However, they also have some drawbacks. For example, ReLU, the most commonly used func tion, is non-differentiable at certain points. This causes a problem during backpropagation, as the process requires the function to be differentiable at all points. The derivatives of these functions at those nondifferentiable points are then artificially defined. In this study, we focus on improving the differentiability of the ReLU function by defining a differentiable piece-wise quadratic approximation of ReLU, which we call Smooth ReLU. We then tune our activation function and evaluate its performance against ReLU using a variety of metrics, on multiple datasets across various networks, commonly used in literature. We further our testing, by com paring Smooth ReLU with seven other widely used activation functions. Our findings suggest that differentiable activation functions can improve the performance of neural networks. By addressing the drawbacks of traditional activation functions, we aim to inspire further research on developing new and improved activation functions and refining existing ones to make them differentiable, thereby advancing the field of Neural Network Optimization.