Abstract:
Neural Networks rely heavily on Activation Functions. These functions determine how the
network responds to its inputs and how it learns. Traditional activation functions, such as the
rectified linear unit (ReLU) and the exponential linear unit (ELU), are effective for many tasks.
However, they also have some drawbacks. For example, ReLU, the most commonly used func
tion, is non-differentiable at certain points. This causes a problem during backpropagation,
as the process requires the function to be differentiable at all points. The derivatives of these
functions at those nondifferentiable points are then artificially defined. In this study, we focus
on improving the differentiability of the ReLU function by defining a differentiable piece-wise
quadratic approximation of ReLU, which we call Smooth ReLU. We then tune our activation
function and evaluate its performance against ReLU using a variety of metrics, on multiple
datasets across various networks, commonly used in literature. We further our testing, by com
paring Smooth ReLU with seven other widely used activation functions. Our findings suggest
that differentiable activation functions can improve the performance of neural networks. By
addressing the drawbacks of traditional activation functions, we aim to inspire further research
on developing new and improved activation functions and refining existing ones to make them
differentiable, thereby advancing the field of Neural Network Optimization.
