kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Beyond speech: leveraging mouse movements for information adaptation in voice interfaces
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0002-8874-6629
Department of Linguistics, University of Potsdam, Potsdam, Germany.ORCID iD: 0000-0002-2686-6887
2025 (English)In: Frontiers in Computer Science, E-ISSN 2624-9898, Vol. 7, article id 1634228Article in journal (Refereed) Published
Abstract [en]

As human speakers naturally adapt their linguistic styles to one another, voice user interfaces that prompt similar linguistic adaptations can augment human-like interaction. In this study, we leverage a corpus of human instructions to model the effectiveness of incremental instruction generation in artificial agents. Participants interacted with agents that guided them in selecting virtual puzzle pieces, varying the amount of information provided in each instruction. Through an empirical examination of the Gricean maxims in utterance construction, our initial perception study highlighted the significance of adaptive instruction generation. By employing mouse movements as a proxy for user understanding, we developed computational models that enabled agents to detect user uncertainty and refine instructions incrementally. Comparing speaker-based and listener-based models, we found that agents encouraging linguistic adaptations were preferred by users. Our findings offer new insights into the value of mouse movements as indicators of user comprehension and introduce a methodological framework for developing adaptive interactive systems that generate instructions dynamically.

Place, publisher, year, edition, pages
Frontiers Media SA , 2025. Vol. 7, article id 1634228
National Category
Human Computer Interaction
Identifiers
URN: urn:nbn:se:kth:diva-374734DOI: 10.3389/fcomp.2025.1634228OAI: oai:DiVA.org:kth-374734DiVA, id: diva2:2023636
Funder
German Research Foundation (DFG)Knut and Alice Wallenberg Foundation
Note

QC 20251221

Available from: 2025-12-19 Created: 2025-12-19 Last updated: 2025-12-21Bibliographically approved

Open Access in DiVA

fulltext(2239 kB)134 downloads
File information
File name FULLTEXT01.pdfFile size 2239 kBChecksum SHA-512
4a1fdd1605c81b3f30de938bb21fe67e915ea246313316666aa4fe10b215a0882611de4693ebe23ac75046fc859e4905ef3062bc2c0b0a127dd451653c0d0d7e
Type fulltextMimetype application/pdf

Other links

Publisher's full text

Authority records

Kontogiorgos, Dimosthenis

Search in DiVA

By author/editor
Kontogiorgos, DimosthenisSchlangen, David
By organisation
Speech, Music and Hearing, TMH
In the same journal
Frontiers in Computer Science
Human Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 1009 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf