kth.sePublikationer
Ändra sökning
Länk till posten
Permanent länk

Direktlänk
Publikationer (10 of 13) Visa alla publikationer
Kalpakchi, D. & Boye, J. (2024). Quinductor: A multilingual data-driven method for generating reading-comprehension questions using Universal Dependencies. Natural Language Engineering, 217-255
Öppna denna publikation i ny flik eller fönster >>Quinductor: A multilingual data-driven method for generating reading-comprehension questions using Universal Dependencies
2024 (Engelska)Ingår i: Natural Language Engineering, ISSN 1351-3249, E-ISSN 1469-8110, s. 217-255Artikel i tidskrift (Refereegranskat) Published
Abstract [en]

We propose a multilingual data-driven method for generating reading comprehension questions using dependency trees. Our method provides a strong, deterministic and inexpensive-to-train baseline for less-resourced languages. While a language-specific corpus is still required, its size is nowhere near those required by modern neural question generation (QG) architectures. Our method surpasses QG baselines previously reported in the literature in terms of automatic evaluation metrics and shows a good performance in terms of human evaluation.

Ort, förlag, år, upplaga, sidor
Cambridge University Press (CUP), 2024
Nyckelord
Natural language generation, Evaluation, Multilinguality, Question generation, Reading comprehension
Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Forskningsämne
Datalogi
Identifikatorer
urn:nbn:se:kth:diva-326862 (URN)10.1017/s1351324923000037 (DOI)000939777300001 ()2-s2.0-85189534486 (Scopus ID)
Forskningsfinansiär
Vinnova, 2019-02997
Anmärkning

QC 20230515

Tillgänglig från: 2023-05-15 Skapad: 2023-05-15 Senast uppdaterad: 2024-04-18Bibliografiskt granskad
Kalpakchi, D. (2023). Ask and distract: Data-driven methods for the automatic generation of multiple-choice reading comprehension questions from Swedish texts. (Doctoral dissertation). KTH Royal Institute of Technology
Öppna denna publikation i ny flik eller fönster >>Ask and distract: Data-driven methods for the automatic generation of multiple-choice reading comprehension questions from Swedish texts
2023 (Engelska)Doktorsavhandling, sammanläggning (Övrigt vetenskapligt)
Alternativ titel[sv]
Fråga och distrahera : Datadrivna metoder för automatisk generering av flervalsfrågor för att bedöma läsförståelse av svenska
Abstract [en]

Multiple choice questions (MCQs) are widely used for summative assessment in many different subjects. The tasks in this format are particularly appealing because they can be graded swiftly and automatically. However, the process of creating MCQs is far from swift or automatic and requires a lot of expertise both in the specific subject and also in test construction.

This thesis focuses on exploring methods for the automatic MCQ generation for assessing the reading comprehension abilities of second-language learners of Swedish. We lay the foundations for the MCQ generation research for Swedish by collecting two datasets of reading comprehension MCQs, and designing and developing methods for generating the whole MCQs or its parts. An important contribution is the methods (which were designed and applied in practice) for the automatic and human evaluation of the generated MCQs.

The best currently available method (as of June 2023) for generating MCQs for assessing reading comprehension in Swedish is ChatGPT (although still only around 60% of generated MCQs were judged acceptable). However, ChatGPT is neither open-source, nor free. The best open-source and free-to-use method is the fine-tuned version of SweCTRL-Mini, a foundational model developed as a part of this thesis. Nevertheless, all explored methods are far from being useful in practice but the reported results provide a good starting point for future research.

Abstract [sv]

Flervalsfrågor används ofta för summativ bedömning i många olika ämnen. Flervalsfrågor är tilltalande eftersom de kan bedömas snabbt och automatiskt. Att skapa flervalsfrågor manuellt går dock långt ifrån snabbt, utan är en process som kräver mycket expertis inom det specifika ämnet och även inom provkonstruktion.

Denna avhandling fokuserar på att utforska metoder för automatisk generering av flervalsfrågor för bedömning av läsförståelse hos andraspråksinlärare av svenska. Vi lägger grunden för forskning om generering av flervalsfrågor för svenska genom att samla in två datamängder bestående av flervalsfrågor som testar just läsförståelse, och genom att utforma och utveckla metoder för att generera hela eller delar av flervalsfrågor. Ett viktigt bidrag är de metoder för automatisk och mänsklig utvärdering av genererade flervalsfrågor som har utvecklats och tillämpats i praktiken.

Den bästa för närvarande tillgängliga metoden (i juni 2023) för att generera flervalsfrågor som testar läsförståelse på svenska är ChatGPT (dock bedömdes endast cirka 60% av de genererade flervalsfrågorna som acceptabla). ChatGPT har dock varken öppen källkod eller är gratis. Den bästa metoden med öppen källkod som är också gratis är den finjusterade versionen av SweCTRL-Mini, en “foundational model” som utvecklats som en del av denna avhandling. Alla utforskade metoder är dock långt ifrån användbara i praktiken, men de rapporterade resultaten ger en bra utgångspunkt för framtida forskning.

Ort, förlag, år, upplaga, sidor
KTH Royal Institute of Technology, 2023. s. viii, 67
Serie
TRITA-EECS-AVL ; 2023:56
Nyckelord
multiple choice questions, question generation, distractor generation, reading comprehension, second-language learners, L2 learning, Natural Language Generation, flervalsfrågor, frågegenerering, distraktorsgenerering, läsförståelse, andraspråkslärande, L2-inlärning, Natural Language Generation
Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Forskningsämne
Tal- och musikkommunikation
Identifikatorer
urn:nbn:se:kth:diva-336531 (URN)978-91-8040-661-1 (ISBN)
Disputation
2023-10-17, F3, Lindstedtsvägen 26, Stockholm, 14:00 (Engelska)
Opponent
Handledare
Anmärkning

QC 20230915

Tillgänglig från: 2023-09-15 Skapad: 2023-09-14 Senast uppdaterad: 2023-09-25Bibliografiskt granskad
Kalpakchi, D. & Boye, J. (2023). Quasi: a synthetic Question-Answering dataset in Swedish using GPT-3 and zero-shot learning. In: Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa): . Paper presented at The 24th Nordic Conference on Computational Linguistics (NoDaLiDa 2023), 22-24 May 2023, Tórshavn, Faroe Islands (pp. 477-491).
Öppna denna publikation i ny flik eller fönster >>Quasi: a synthetic Question-Answering dataset in Swedish using GPT-3 and zero-shot learning
2023 (Engelska)Ingår i: Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), 2023, s. 477-491Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

This paper describes the creation and evaluation of a synthetic dataset of Swedish multiple-choice questions (MCQs) for reading comprehension using GPT-3. Although GPT-3 is trained mostly on English data, with only 0.11% of Swedish texts in its training material, the model still managed to generate MCQs in Swedish. About 44% of the generated MCQs turned out to be of sufficient quality, i.e.\ they were grammatically correct and relevant, with exactly one answer alternative being correct and the others being plausible but wrong. We provide a detailed analysis of the errors and shortcomings of the rejected MCQs, as well an analysis of the level of difficulty of the accepted MCQs. In addition to giving insights into GPT-3, the synthetic dataset could be used for training and evaluation of special-purpose MCQ-generating models.

Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Forskningsämne
Datalogi
Identifikatorer
urn:nbn:se:kth:diva-327972 (URN)
Konferens
The 24th Nordic Conference on Computational Linguistics (NoDaLiDa 2023), 22-24 May 2023, Tórshavn, Faroe Islands
Anmärkning

QC 20230602

Tillgänglig från: 2023-06-02 Skapad: 2023-06-02 Senast uppdaterad: 2023-09-14Bibliografiskt granskad
Kalpakchi, D. & Boye, J. (2022). Automatically generating question-answer pairs for assessing basic reading comprehension in Swedish. In: : . Paper presented at The 9th Swedish Language Technology Conference (SLTC 2022), Stockholm, Sweden, 23–25 November 2022.
Öppna denna publikation i ny flik eller fönster >>Automatically generating question-answer pairs for assessing basic reading comprehension in Swedish
2022 (Engelska)Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

This paper presents an evaluation of the quality of automatically generated reading comprehension questions from Swedish text, using the Quinductor method. This method is a light-weight, data-driven but non-neural method for automatic question generation (QG). The evaluation shows that Quinductor is a viable QG method that can provide a strong baseline for neural-network-based QG methods. 

Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Identifikatorer
urn:nbn:se:kth:diva-326889 (URN)10.48550/arXiv.2211.15568 (DOI)
Konferens
The 9th Swedish Language Technology Conference (SLTC 2022), Stockholm, Sweden, 23–25 November 2022
Forskningsfinansiär
Vinnova, 2019-02997
Anmärkning

QC 20230515

Tillgänglig från: 2023-05-15 Skapad: 2023-05-15 Senast uppdaterad: 2023-09-14Bibliografiskt granskad
Willemsen, B., Kalpakchi, D. & Skantze, G. (2022). Collecting Visually-Grounded Dialogue with A Game Of Sorts. In: Calzolari, N Bechet, F Blache, P Choukri, K Cieri, C Declerck, T Goggi, S Isahara, H Maegaard, B Mazo, H Odijk, H Piperidis, S (Ed.), Proceedings of the 13th Conference on Language Resources and Evaluation: . Paper presented at 13th Conference on Language Resources and Evaluation, 20-25 June, Marseille, France, 2022 (pp. 2257-2268). European Language Resources Association (ELRA)
Öppna denna publikation i ny flik eller fönster >>Collecting Visually-Grounded Dialogue with A Game Of Sorts
2022 (Engelska)Ingår i: Proceedings of the 13th Conference on Language Resources and Evaluation / [ed] Calzolari, N Bechet, F Blache, P Choukri, K Cieri, C Declerck, T Goggi, S Isahara, H Maegaard, B Mazo, H Odijk, H Piperidis, S, European Language Resources Association (ELRA) , 2022, s. 2257-2268Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

An idealized, though simplistic, view of the referring expression production and grounding process in (situated) dialogue assumes that a speaker must merely appropriately specify their expression so that the target referent may be successfully identified by the addressee. However, referring in conversation is a collaborative process that cannot be aptly characterized as an exchange of minimally-specified referring expressions. Concerns have been raised regarding assumptions made by prior work on visually-grounded dialogue that reveal an oversimplified view of conversation and the referential process. We address these concerns by introducing a collaborative image ranking task, a grounded agreement game we call “A Game Of Sorts”. In our game, players are tasked with reaching agreement on how to rank a set of images given some sorting criterion through a largely unrestricted, role-symmetric dialogue. By putting emphasis on the argumentation in this mixed-initiative interaction, we collect discussions that involve the collaborative referential process. We describe results of a small-scale data collection experiment with the proposed task. All discussed materials, which includes the collected data, the codebase, and a containerized version of the application, are publicly available.

Ort, förlag, år, upplaga, sidor
European Language Resources Association (ELRA), 2022
Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Identifikatorer
urn:nbn:se:kth:diva-323116 (URN)000889371702039 ()2-s2.0-85144393073 (Scopus ID)
Konferens
13th Conference on Language Resources and Evaluation, 20-25 June, Marseille, France, 2022
Anmärkning

Part of proceedings: ISBN 9791095546726

QC 20230125

Tillgänglig från: 2023-01-16 Skapad: 2023-01-16 Senast uppdaterad: 2023-10-30Bibliografiskt granskad
Kalpakchi, D. & Boye, J. (2022). Textinator: an Internationalized Tool for Annotation and Human Evaluation in Natural Language Processing and Generation. In: Calzolari, N Bechet, F Blache, P Choukri, K Cieri, C Declerck, T Goggi, S Isahara, H Maegaard, B Mazo, H Odijk, H Piperidis, S (Ed.), LREC 2022: Thirteen International Conference On Language Resources And Evaluation. Paper presented at 13th International Conference on Language Resources and Evaluation (LREC), JUN 20-25, 2022, Marseille, FRANCE (pp. 856-866). European Language Resources Association (ELRA)
Öppna denna publikation i ny flik eller fönster >>Textinator: an Internationalized Tool for Annotation and Human Evaluation in Natural Language Processing and Generation
2022 (Engelska)Ingår i: LREC 2022: Thirteen International Conference On Language Resources And Evaluation / [ed] Calzolari, N Bechet, F Blache, P Choukri, K Cieri, C Declerck, T Goggi, S Isahara, H Maegaard, B Mazo, H Odijk, H Piperidis, S, European Language Resources Association (ELRA) , 2022, s. 856-866Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

We release an internationalized annotation and human evaluation bundle, called Textinator, along with documentation and video tutorials. Textinator allows annotating data for a wide variety of NLP tasks, and its user interface is offered in multiple languages, lowering the entry threshold for domain experts. The latter is, in fact, quite a rare feature among the annotation tools, that allows controlling for possible unintended biases introduced due to hiring only English-speaking annotators. We illustrate the rarity of this feature by presenting a thorough systematic comparison of Textinator to previously published annotation tools along 9 different axes (with internationalization being one of them). To encourage researchers to design their human evaluation before starting to annotate data, Textinator offers an easy-to-use tool for human evaluations allowing importing surveys with potentially hundreds of evaluation items in one click. We finish by presenting several use cases of annotation and evaluation projects conducted using pre-release versions of Textinator. The presented use cases do not represent Textinator's full annotation or evaluation capabilities, and interested readers are referred to the online documentation for more information.

Ort, förlag, år, upplaga, sidor
European Language Resources Association (ELRA), 2022
Nyckelord
annotation tool, human evaluation tool, natural language processing, natural language generation
Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Identifikatorer
urn:nbn:se:kth:diva-324335 (URN)10.5281/zenodo.6497334 (DOI)000889371700090 ()2-s2.0-85144462359 (Scopus ID)
Konferens
13th International Conference on Language Resources and Evaluation (LREC), JUN 20-25, 2022, Marseille, FRANCE
Anmärkning

QC 20230228

Tillgänglig från: 2023-02-28 Skapad: 2023-02-28 Senast uppdaterad: 2023-09-14Bibliografiskt granskad
Kalpakchi, D. & Boye, J. (2021). BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset. In: Proceedings of the 14th International Conference on Natural Language Generation: . Paper presented at 14th International Conference on Natural Language Generation, INLG 2021, Virtual/Online, 20-24 September 2021 (pp. 387-403).
Öppna denna publikation i ny flik eller fönster >>BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset
2021 (Engelska)Ingår i: Proceedings of the 14th International Conference on Natural Language Generation, 2021, s. 387-403Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

An important part when constructing multiple-choice questions (MCQs) for reading comprehension assessment are the distractors, the incorrect but preferably plausible answer options. In this paper, we present a new BERT-based method for automatically generating distractors using only a small-scale dataset. We also release a new such dataset of Swedish MCQs (used for training the model), and propose a methodology for assessing the generated distractors. Evaluation shows that from a student's perspective, our method generated one or more plausible distractors for more than 50% of the MCQs in our test set. From a teacher's perspective, about 50% of the generated distractors were deemed appropriate. We also do a thorough analysis of the results

Nyckelord
Multiple-choice questions, Reading comprehension, Small scale, Student perspectives, Swedishs, Teachers', Test sets
Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Forskningsämne
Datalogi
Identifikatorer
urn:nbn:se:kth:diva-302480 (URN)2-s2.0-85123291566 (Scopus ID)
Konferens
14th International Conference on Natural Language Generation, INLG 2021, Virtual/Online, 20-24 September 2021
Forskningsfinansiär
Vinnova, 2019-02997
Anmärkning

Part of proceedings: ISBN 978-1-954085-51-0

QC 20220301

Tillgänglig från: 2021-09-24 Skapad: 2021-09-24 Senast uppdaterad: 2023-09-14Bibliografiskt granskad
Kalpakchi, D. & Boye, J. (2021). Minor changes make a difference: a case study on the consistency of UD-based dependency parsers. In: Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021): . Paper presented at UDW 2021 - 5th Workshop on Universal Dependencies, Proceedings - To be held as part of SyntaxFest 2021, Sofia, 21-25 March 2021 (pp. 96-108). Association for Computational Linguistics (ACL)
Öppna denna publikation i ny flik eller fönster >>Minor changes make a difference: a case study on the consistency of UD-based dependency parsers
2021 (Engelska)Ingår i: Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021), Association for Computational Linguistics (ACL) , 2021, s. 96-108Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

Many downstream applications are using dependency trees, and are thus relying on dependencyparsers producing correct, or at least consistent, output. However, dependency parsers are trainedusing machine learning, and are therefore susceptible to unwanted inconsistencies due to biasesin the training data. This paper explores the effects of such biases in four languages – English,Swedish, Russian, and Ukrainian – though an experiment where we study the effect of replacingnumerals in sentences. We show that such seemingly insignificant changes in the input can causelarge differences in the output, and suggest that data augmentation can remedy the problems.

Ort, förlag, år, upplaga, sidor
Association for Computational Linguistics (ACL), 2021
Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Forskningsämne
Datalogi
Identifikatorer
urn:nbn:se:kth:diva-326888 (URN)2-s2.0-85138675937 (Scopus ID)
Konferens
UDW 2021 - 5th Workshop on Universal Dependencies, Proceedings - To be held as part of SyntaxFest 2021, Sofia, 21-25 March 2021
Forskningsfinansiär
Vinnova, 2019-02997
Anmärkning

Part of proceedings ISBN 978-195591717-9 

QC 20230515

Tillgänglig från: 2023-05-15 Skapad: 2023-05-15 Senast uppdaterad: 2023-09-14Bibliografiskt granskad
Kalpakchi, D. & Boye, J. (2020). UDon2: a library for manipulating Universal Dependencies trees. In: Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020): . Paper presented at 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), 8-13 December 2020 (pp. 120-125).
Öppna denna publikation i ny flik eller fönster >>UDon2: a library for manipulating Universal Dependencies trees
2020 (Engelska)Ingår i: Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), 2020, s. 120-125Konferensbidrag, Poster (med eller utan abstract) (Refereegranskat)
Abstract [en]

UDon2 is an open-source library for manipulating dependency trees represented in the CoNLL-U format. The library is compatible with the Universal Dependencies. UDon2 is aimed at developers of downstream Natural Language Processing applications that require manipulating dependency trees on the sentence level (in addition to other available tools geared towards working with treebanks).

Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Identifikatorer
urn:nbn:se:kth:diva-288878 (URN)
Konferens
28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), 8-13 December 2020
Forskningsfinansiär
Vinnova
Anmärkning

QC 20210115

Tillgänglig från: 2021-01-14 Skapad: 2021-01-14 Senast uppdaterad: 2023-09-14Bibliografiskt granskad
Kalpakchi, D. & Boye, J. (2019). SpaceRefNet: a neural approach to spatial reference resolution in a real city environment. In: Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue: . Paper presented at 20th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2019, 11-13 September 2019, Stockholm, Sweden (pp. 422-431). Association for Computational Linguistics
Öppna denna publikation i ny flik eller fönster >>SpaceRefNet: a neural approach to spatial reference resolution in a real city environment
2019 (Engelska)Ingår i: Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, Association for Computational Linguistics , 2019, s. 422-431Konferensbidrag, Muntlig presentation med publicerat abstract (Refereegranskat)
Abstract [en]

Adding interactive capabilities to pedestrian wayfinding systems in the form of spoken dialogue will make them more natural to humans. Such an interactive wayfinding system needs to continuously understand and interpret pedestrian’s utterances referring to the spatial context. Achieving this requires the system to identify exophoric referring expressions in the utterances, and link these expressions to the geographic entities in the vicinity. This exophoric spatial reference resolution problem is difficult, as there are often several dozens of candidate referents. We present a neural network-based approach for identifying pedestrian’s references (using a network called RefNet) and resolving them to appropriate geographic objects (using a network called SpaceRefNet). Both methods show promising results beating the respective baselines and earlier reported results in the literature.

Ort, förlag, år, upplaga, sidor
Association for Computational Linguistics, 2019
Nationell ämneskategori
Språkteknologi (språkvetenskaplig databehandling)
Identifikatorer
urn:nbn:se:kth:diva-262883 (URN)10.18653/v1/w19-5949 (DOI)000591510500049 ()2-s2.0-85091595033 (Scopus ID)
Konferens
20th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2019, 11-13 September 2019, Stockholm, Sweden
Anmärkning

QC 20210914

Tillgänglig från: 2019-10-22 Skapad: 2019-10-22 Senast uppdaterad: 2022-06-26Bibliografiskt granskad
Organisationer
Identifikatorer
ORCID-id: ORCID iD iconorcid.org/0000-0001-7327-3059

Sök vidare i DiVA

Visa alla publikationer