Investigation of activation functions in deep belief network
MetadataShow full item record
© 2017 IEEE. Deep Belief Network (DBN) is made up of stacked Restricted Boltzmann Machine layers associated with global weight fine-tuning for pattern recognition. However, DBN suffers from vanishing gradient problem due to the saturation characteristic of activation function. Therefore, the selection of activation function in DBN is critical to reduce the network complexity and improve performance of pattern recognition. Unsaturated activation functions such as rectified linear unit and leaky rectified linear unit were recently proposed to avoid the effect of vanishing gradient for a deep learning neural network. In this paper, we investigated the network performance with both saturated and unsaturated activation functions. Besides that, the randomization of training samples would significantly improve the performance of DBN. The experimental results showed that hyperbolic tangent activation function achieved the lowest error rate which is 1.99% on MNIST handwritten digit dataset.
Showing items related by title, author, creator and subject.
Meir, K.; Gaffney, E.; Simeon-Dubach, D.; Ravid, R.; Watson, P.; Schacter, B.; Morente, M.; Bjugn, R.; Clark, B.; De Blasio, P.; Carpenter, J.; Deschenes, M.; Devereux, L.; Dhir, R.; Goebell, P.; Grizzle, W.; Hainaut, P.; Mes-Masson, A.; Miranda, L.; Parry-Jones, A.; Riegman, P.; Casali-Da-Rocha, J.; Soares, F.; Vaught, J.; Zeps, Nikolajs (2011)The biobanking literature frequently addresses donor and societal issues surrounding biobanking, but the biobanker's perspective is rarely highlighted. While not comprehensive, this article offers an overview of the human ...
Impaired gamma-band activity during perceptual organization in adults with autism spectrum disorders: Evidence for dysfunctional network activity in frontal-posterior corticesSun, L.; Grützner, C.; Bölte, Sven; Wibral, M.; Tozman, T.; Schlitt, S.; Poustka, F.; Singer, W.; Freitag, C.; Uhlhaas, P. (2012)Current theories of the pathophysiology of autism spectrum disorders (ASD) have focused on abnormal temporal coordination of neural activity in cortical circuits as a core impairment of the disorder. In the current study, ...
Luo, R.; Xu, Honglei; Wang, W.; Sun, Jie; Xu, W. (2016)The classical analysis of asymptotical and exponential stability of neural networks needs assumptions on the existence of a positive Lyapunov function V and on the strict negativity of the function dV=dt, which often come ...