kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Exploring Latent Sign Language Representations with Isolated Signs, Sentences and In-the-Wild Data
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0009-0006-4548-4434
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0009-0002-2374-0856
Department of Linguistics, Stockholm University, Sweden.ORCID iD: 0000-0002-0612-6304
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0003-1399-6604
2024 (English)In: 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources, sign-lang@LREC-COLING 2024, Association for Computational Linguistics (ACL) , 2024, p. 219-224Conference paper, Published paper (Refereed)
Abstract [en]

Unsupervised representation learning offers a promising way of utilising large unannotated sign language resources found on the Internet. In this paper, a representation learning model, VQ-VAE, is trained to learn a codebook of motion primitives from sign language data. For training, we use isolated signs and sentences from a sign language dictionary. Three models are trained: one on isolated signs, one on sentences, and one mixed model. We test these models by comparing how well they are able to reconstruct held-out data from the dictionary, as well as an in-the-wild dataset based on sign language videos from YouTube. These data are characterized by less formal and more expressive signing than the dictionary items. Results show that the isolated sign model yields considerably higher reconstruction loss for the YouTube dataset, while the sentence model performs the best on this data. Further, an analysis of codebook usage reveals that the set of codes used by isolated signs and sentences differ significantly. In order to further understand the different characters of the datasets, we carry out an analysis of the velocity profiles, which reveals that signing data in-the-wild has a much higher average velocity than dictionary signs and sentences. We believe these differences also explain the large differences in reconstruction loss observed.

Place, publisher, year, edition, pages
Association for Computational Linguistics (ACL) , 2024. p. 219-224
Keywords [en]
Pose Codebook, Representation Learning, sign language data, VQ-VAE
National Category
General Language Studies and Linguistics
Identifiers
URN: urn:nbn:se:kth:diva-350726Scopus ID: 2-s2.0-85197480349OAI: oai:DiVA.org:kth-350726DiVA, id: diva2:1884692
Conference
11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources, sign-lang@LREC-COLING 2024, Torino, Italy, May 25 2024
Projects
signbot
Note

Part of ISBN 9782493814302

QC 20240719

Available from: 2024-07-17 Created: 2024-07-17 Last updated: 2024-10-23Bibliographically approved

Open Access in DiVA

fulltext(540 kB)104 downloads
File information
File name FULLTEXT01.pdfFile size 540 kBChecksum SHA-512
d7e1052583e3570f573b4689bc862b0e4a079978205577d133cee6ef6ef6e434a30c8b7f4c465032020c1a6ff5f7ade0a03b80d99ccd31cddfdade69da6b4d4f
Type fulltextMimetype application/pdf

Scopus

Authority records

Malmberg, FredrikKlezovich, AnnaBeskow, Jonas

Search in DiVA

By author/editor
Malmberg, FredrikKlezovich, AnnaMesch, JohannaBeskow, Jonas
By organisation
Speech, Music and Hearing, TMH
General Language Studies and Linguistics

Search outside of DiVA

GoogleGoogle Scholar
Total: 104 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 1272 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf