EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks.

Clicks: 279
ID: 265167
2021
Article Quality & Performance Metrics
Overall Quality
Not rated
Combines reader engagement with the AI quality analysis. This article has not been analysed, so there is no overall score — reader engagement is measured and shown alongside.
AI Quality Assessment
Not analyzed
Readership in this journal
Emerging

Ranked #32 of 56 articles by views in IEEE Transactions on Neural Networks and Learning Systems

Most read Least read

Bar heights use a square-root scale.

Mint this article as an NFT
Not yet minted

Create a permanent, verifiable on-chain record of this article on the Scimatic Network. The NFT is held in your Journament account, and you can withdraw it to your own wallet at any time.

5 SUSD one-off · no wallet required
Sign in to mint
Abstract
Dropout is a well-known regularization method by sampling a sub-network from a larger deep neural network and training different sub-networks on different subsets of the data. Inspired by the dropout concept, we propose EDropout as an energy-based framework for pruning neural networks in classification tasks. In this approach, a set of binary pruning state vectors (population) represents a set of corresponding sub-networks from an arbitrary original neural network. An energy loss function assigns a scalar energy loss value to each pruning state. The energy-based model (EBM) stochastically evolves the population to find states with lower energy loss. The best pruning state is then selected and applied to the original network. Similar to dropout, the kept weights are updated using backpropagation in a probabilistic model. The EBM again searches for better pruning states and the cycle continuous. This procedure is a switching between the energy model, which manages the pruning states, and the probabilistic model, which updates the kept weights, in each iteration. The population can dynamically converge to a pruning state. This can be interpreted as dropout leading to pruning the network. From an implementation perspective, unlike most of the pruning methods, EDropout can prune neural networks without manually modifying the network architecture code. We have evaluated the proposed method on different flavors of ResNets, AlexNet, l₁ pruning, ThinNet, ChannelNet, and SqueezeNet on the Kuzushiji, Fashion, CIFAR-10, CIFAR-100, Flowers, and ImageNet data sets, and compared the pruning rate and classification performance of the models. The networks trained with EDropout on average achieved a pruning rate of more than 50% of the trainable parameters with approximately <5% and <1% drop of Top-1 and Top-5 classification accuracy, respectively.
Reference Key
salehinejad2021edropoutieee Use this key to autocite in the manuscript while using SciMatic Manuscript Manager or Thesis Manager
Authors Salehinejad, Hojjat;Valaee, Shahrokh;
Journal IEEE Transactions on Neural Networks and Learning Systems
Year 2021
DOI
10.1109/TNNLS.2021.3069970
URL
Keywords
phylogeny ani chinese cabbage (brassica rapa l. ssp. pekinensis) pectobacterium polaris bacterial soft rot phenotypic characteristics specific pcr/qpcr detection

Citations

No citations found. To add a citation, contact the admin at info@scimatic.org

Comments

No comments yet. Be the first to comment on this article.