kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
How "open" are the conversations with open-domain chatbots?: A proposal for Speech Event based evaluation
Univ Ghent, Dept Translat Interpreting & Commun, LT3, Ghent, Belgium..
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0002-8579-1790
2021 (English)In: SIGDIAL 2021: 22Nd Annual Meeting Of The Special Interest Group On Discourse And Dialogue (Sigdial 2021), ASSOC COMPUTATIONAL LINGUISTICS , 2021, p. 392-402Conference paper, Published paper (Refereed)
Abstract [en]

Open-domain chatbots are supposed to converse freely with humans without being restricted to a topic, task or domain. However, the boundaries and/or contents of open-domain conversations are not clear. To clarify the boundaries of "openness", we conduct two studies: First, we classify the types of "speech events" encountered in a chatbot evaluation data set (i.e., Meena by Google) and find that these conversations mainly cover the "small talk" category and exclude the other speech event categories encountered in real life human-human communication. Second, we conduct a small-scale pilot study to generate online conversations covering a wider range of speech event categories between two humans vs. a human and a state-of-the-art chatbot (i.e., Blender by Facebook). A human evaluation of these generated conversations indicates a preference for human-human conversations, since the human-chatbot conversations lack coherence in most speech event categories. Based on these results, we suggest (a) using the term "small talk" instead of "opendomain" for the current chatbots which are not that "open" in terms of conversational abilities yet, and (b) revising the evaluation methods to test the chatbot conversations against other speech events.

Place, publisher, year, edition, pages
ASSOC COMPUTATIONAL LINGUISTICS , 2021. p. 392-402
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-304763ISI: 000707001800041Scopus ID: 2-s2.0-85137760284OAI: oai:DiVA.org:kth-304763DiVA, id: diva2:1610962
Conference
22nd Annual Meeting of the Special-Interest-Group-on-Discourse-and-Dialogue (SIGDIAL), JUL 29-31, 2021, Singapore, SINGAPORE
Note

Part of proceedings: ISBN 978-1-954085-81-7

QC 20230117

Available from: 2021-11-12 Created: 2021-11-12 Last updated: 2023-07-14Bibliographically approved

Open Access in DiVA

No full text in DiVA

Scopus

Authority records

Skantze, Gabriel

Search in DiVA

By author/editor
Skantze, Gabriel
By organisation
Speech, Music and Hearing, TMH
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 92 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf