How can deep rectifier networks achieve linear separability and preserve distances?
MetadataShow full item record
This paper investigates how hidden layers of deep rectifier networks are capable of transforming two or more pattern sets to be linearly separable while preserving the distances with a guaranteed degree, and proves the universal classification power of such distance preserving rectifier networks. Through the nearly isometric nonlinear transformation in the hidden layers, the margin of the linear separating plane in the output layer and the margin of the nonlinear separating boundary in the original data space can be closely related so that the maximum margin classification in the input data space can be achieved approximately via the maximum margin linear classifiers in the output layer. The generalization performance of such distance preserving deep rectifier neural networks can be well justified by the distance-preserving properties of their hidden layers and the maximum margin property of the linear classifiers in the output layer.
Showing items related by title, author, creator and subject.
An, Senjian; Hayat, M.; Khan, S.; Bennamoun, M.; Boussaid, F.; Sohel, F. (2015)© 2015 IEEE. To find the optimal nonlinear separating boundary with maximum margin in the input data space, this paper proposes Contractive Rectifier Networks (CRNs), wherein the hidden-layer transformations are restricted ...
An, Senjian; Ke, Q.; Bennamoun, M.; Boussaid, F.; Sohel, F. (2015)© Springer International Publishing Switzerland 2015. In this paper we introduce sign constrained rectifier networks (SCRN), demonstrate their universal classification power and illustrate their applications to pattern ...
Lau, M.; Lim, Hann (2017)© 2017 IEEE. Deep Belief Network (DBN) is made up of stacked Restricted Boltzmann Machine layers associated with global weight fine-tuning for pattern recognition. However, DBN suffers from vanishing gradient problem due ...