Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
The Convex Information Bottleneck Lagrangian
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering.ORCID iD: 0000-0002-0862-1333
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering.ORCID iD: 0000-0001-9307-484X
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering. KTH, Superseded Departments (pre-2005), Signals, Sensors and Systems.ORCID iD: 0000-0002-7926-5081
2020 (English)In: Entropy, ISSN 1099-4300, E-ISSN 1099-4300, Vol. 22, no 1, article id 98Article in journal (Refereed) Published
Abstract [en]

The information bottleneck (IB) problem tackles the issue of obtaining relevant compressedrepresentations T of some random variable X for the task of predicting Y. It is defined as a constrainedoptimization problem that maximizes the information the representation has about the task, I(T;Y) ,while ensuring that a certain level of compression r is achieved (i.e., I(X;T) ≤ r). For practical reasons,the problem is usually solved by maximizing the IB Lagrangian for many values of the Lagrange multiplier. Then, the curve of maximal I(T;Y) for a givenI(X;T) is drawn anda representation with the desired predictability and compression is selected. It is known when Yis a deterministic function of X, the IB curve cannot be explored and another Lagrangian has beenproposed to tackle this problem: the squared IB Lagrangian. In this paper, we (i) present a general family of Lagrangians which allow for the exploration of the IBcurve in all scenarios; (ii) provide the exact one-to-one mapping between the Lagrange multiplierand the desired compression rate r for known IB curve shapes; and (iii) show we can approximatelyobtain a specific compression level with the convex IB Lagrangian for both known and unknown IBcurve shapes. This eliminates the burden of solving the optimization problem for many values of theLagrange multiplier. That is, we prove that we can solve the original constrained problem with asingle optimization.

Place, publisher, year, edition, pages
MDPI, 2020. Vol. 22, no 1, article id 98
Keywords [en]
information bottleneck; representation learning; mutual information; optimization
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:kth:diva-266691DOI: 10.3390/e22010098Scopus ID: 2-s2.0-85078523691OAI: oai:DiVA.org:kth-266691DiVA, id: diva2:1386077
Note

QC 20200120

Available from: 2020-01-16 Created: 2020-01-16 Last updated: 2020-02-04Bibliographically approved

Open Access in DiVA

fulltext(1499 kB)11 downloads
File information
File name FULLTEXT01.pdfFile size 1499 kBChecksum SHA-512
80554270a836881dd118136da93f367ff79c91b463f48e09aabb7c46cec282f73b46a3ae5629533c7cf64d975641e7d87eeb9cd49a6fd4f18af462bc549940fa
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopushttps://www.mdpi.com/1099-4300/22/1/98

Authority records BETA

Rodríguez Gálvez, BorjaThobaben, RagnarSkoglund, Mikael

Search in DiVA

By author/editor
Rodríguez Gálvez, BorjaThobaben, RagnarSkoglund, Mikael
By organisation
Information Science and EngineeringSignals, Sensors and Systems
In the same journal
Entropy
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 11 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 48 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf