kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Q-S5 Towards Quantized State Space Models
University of Groningen, Groningen, The Netherlands; Intel Labs, Munich, Germany.ORCID iD: 0000-0002-2272-315X
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Computational Science and Technology (CST).ORCID iD: 0000-0001-6012-7415
University of Cambridge, Cambridge, UK.
Ludwig-Maximilians-Universität München, Munich, Germany; Intel Labs, Munich, Germany.ORCID iD: 0000-0002-5682-627X
2024 (English)In: Next Generation of Sequence Modeling Architectures Workshop, 2024Conference paper, Oral presentation with published abstract (Refereed)
Abstract [en]

In the quest for next-generation sequence modeling architectures, State Space Models (SSMs) have emerged as a potent alternative to transformers, particularly for their computational efficiency and suitability for dynamical systems. This paper investigates the effect of quantization on the S5 model to understand its impact on model performance and to facilitate its deployment to edge and resource-constrained platforms. Using quantization-aware training (QAT) and post-training quantization (PTQ), we systematically evaluate the quantization sensitivity of SSMs across different tasks like dynamical systems modeling, Sequential MNIST (sMNIST) and most of the Long Range Arena (LRA). We present fully quantized S5 models whose test accuracy drops less than 1% on sMNIST and most of the LRA. We find that performance on most tasks degrades significantly for recurrent weights below 8-bit precision, but that other components can be compressed further without significant loss of performance. Our results further show that PTQ only performs well on language-based LRA tasks whereas all others require QAT. Our investigation provides necessary insights for the continued development of efficient and hardware-optimized SSMs.

Place, publisher, year, edition, pages
2024.
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-370474OAI: oai:DiVA.org:kth-370474DiVA, id: diva2:2001103
Conference
ICML 2024 - The Forty-First International Conference on Machine Learning, Vienna, Austria, July 21-27, 2024
Note

QC 20250926

Available from: 2025-09-25 Created: 2025-09-25 Last updated: 2025-09-26Bibliographically approved
In thesis
1. Neuromorphic computation in space and time: On first-principles approaches to computation in mixed-signal neural networks
Open this publication in new window or tab >>Neuromorphic computation in space and time: On first-principles approaches to computation in mixed-signal neural networks
2025 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Digital computers have advanced to rival human-level intelligence across creative reasoning and complex problem-solving tasks. Yet, theoretical comparisons with unconventional computational substrates suggest we have only scratched the surface of computational potential. Biological nervous systems have long been studied for their efficiency and robustness, inspiring the invention of neuromorphic hardware. Neuromorphic systems have yet to outperform digital computers, likely due to our limited understanding of the governing computational principles.

This work investigates these computational principles using two modes of inquiry. First, axioms for mixed-signal neural networks are studied by induction as necessary conditions for provably correct neuromorphic computations. Part of the axioms are applied in the Neuromorphic Intermediate Representation, a set of neuromorphic primitives, which is demonstrated to work across more than 12 neuromorphic software and hardware platforms. A second inquiry is made into geometric approaches to event-based vision by deduction. By establishing a direct relationship between neuromorphic primitives and signal transformations, it is demonstrated how neural networks can be imbued with covariance properties that enable them to outperform conventional networks of similar complexity in object tracking tasks. The complementary inductive-deductive approaches provide a more complete lens from which to understand and implement neuromorphic computation.

Additionally, a chapter is dedicated to several openly accessible software projects for evaluating neuromorphic systems on commodity hardware. Apart from being the backbone for the research in this thesis, the accessibility and reproducibility may propagate the research and catalyze community efforts.

Finally, the thesis concludes with a discussion on the broader implications of the above findings and the future trajectory of neuromorphic computation.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2025. p. xi, 81
Series
TRITA-EECS-AVL ; 2025:85
Keywords
Neuromorphic computing, mixed-signal neural networks, event-based vision, scale-space theory, computational primitives, covariance, computational models.
National Category
Computer and Information Sciences
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-370478 (URN)978-91-8106-406-3 (ISBN)
Public defence
2025-10-17, https://kth-se.zoom.us/j/68948445390?pwd=DcZEttQgMF9NzidYOSYLzpypopsXVG.1, D3, Lindstedtsvägen 5, KTH, Stockholm, Stockholm, 14:00 (English)
Opponent
Supervisors
Note

QC 20250926

Available from: 2025-09-26 Created: 2025-09-25 Last updated: 2025-10-13Bibliographically approved

Open Access in DiVA

fulltext(2223 kB)48 downloads
File information
File name FULLTEXT01.pdfFile size 2223 kBChecksum SHA-512
737c69a00485c46ce7759a88c4c4285f10d4e257b878b85b24ad2c16cb7537da2cd4b74fa1b6e74b9057b859498eb97fb1deaa624300a2fb171297c5b55371fd
Type fulltextMimetype application/pdf

Authority records

Pedersen, Jens Egholm

Search in DiVA

By author/editor
Abreu, StevenPedersen, Jens EgholmPierro, Alessandro
By organisation
Computational Science and Technology (CST)
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 553 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf