kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Relu Dense Layer To Improve The Performance Of Neural Networks
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering.ORCID iD: 0000-0002-8534-7622
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering.
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering.ORCID iD: 0000-0002-7926-5081
KTH, School of Electrical Engineering and Computer Science (EECS), Centres, ACCESS Linnaeus Centre. KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering.ORCID iD: 0000-0003-2638-6047
2021 (English)In: 2021 IEEE International Conference On Acoustics, Speech And Signal Processing (ICASSP 2021), Institute of Electrical and Electronics Engineers (IEEE) , 2021, p. 2810-2814Conference paper, Published paper (Refereed)
Abstract [en]

We propose ReDense as a simple and low complexity way to improve the performance of trained neural networks. We use a combination of random weights and rectified linear unit (ReLU) activation function to add a ReLU dense (ReDense) layer to the trained neural network such that it can achieve a lower training loss. The lossless flow property (LFP) of ReLU is the key to achieve the lower training loss while keeping the generalization error small. ReDense does not suffer from vanishing gradient problem in the training due to having a shallow structure. We experimentally show that ReDense can improve the training and testing performance of various neural network architectures with different optimization loss and activation functions. Finally, we test ReDense on some of the state-of-the-art architectures and show the performance improvement on benchmark.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE) , 2021. p. 2810-2814
Keywords [en]
Rectified linear unit, random weights, deep neural network
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-305410DOI: 10.1109/ICASSP39728.2021.9414269ISI: 000704288403013Scopus ID: 2-s2.0-85115078893OAI: oai:DiVA.org:kth-305410DiVA, id: diva2:1615887
Conference
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), JUN 06-11, 2021, ELECTR NETWORK
Note

Part of proceedings: ISBN 978-1-7281-7605-5, QC 20230118

Available from: 2021-12-01 Created: 2021-12-01 Last updated: 2023-01-18Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Javid, Alireza M.Das, SandipanSkoglund, MikaelChatterjee, Saikat

Search in DiVA

By author/editor
Javid, Alireza M.Das, SandipanSkoglund, MikaelChatterjee, Saikat
By organisation
Information Science and EngineeringACCESS Linnaeus Centre
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 49 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf