kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Human vs LLM: A Comparative Performance Analysis in a Custom Strategic Deduction Game
KTH, School of Electrical Engineering and Computer Science (EECS). University of Twente, Enschede, Netherlands.
KTH, School of Electrical Engineering and Computer Science (EECS). University of Twente, Enschede, Netherlands.
2025 (English)In: Americas Conference on Information Systems, AMCIS 2025, Association for Information Systems , 2025, Vol. 5, p. 3468-3477Conference paper, Published paper (Refereed)
Abstract [en]

Understanding how large language models perform relative to humans in socially interactive, deduction-based tasks is vital for advancing AI applications. This study compares the performance of human players and GPT-40 in Guess vs. AI, a custom strategic deduction game. Drawing on data from 85 completed games, the AI-opponent achieved a significantly higher win rate than human players (63.5%, p = 0.009) and required fewer questions to identify the target (humans: 17, AI-opponent: 9). These findings highlight GPT-40's strengths in systematic reasoning, pattern recognition and efficient decision-making. While showcasing the potential of large language models in structured deduction scenarios, they also emphasize the need for further research into AI adaptability in more socially complex tasks. Future directions include expanding demographic diversity, exploring additional game formats or different large language models and investigating potential human-AI collaborations rather than strictly competitive environments.

Place, publisher, year, edition, pages
Association for Information Systems , 2025. Vol. 5, p. 3468-3477
Keywords [en]
AI vs Human Performance, Deductive Reasoning, Generative AI, Human-AI Interaction, Large Language Models, Prompt Engineering, Strategy Games
National Category
Information Systems
Identifiers
URN: urn:nbn:se:kth:diva-385580Scopus ID: 2-s2.0-105025162305OAI: oai:DiVA.org:kth-385580DiVA, id: diva2:2086812
Conference
2025 Americas Conference on Information Systems, AMCIS 2025, Montreal, Canada, August 14-16, 2025
Note

Part of ISBN 9798331327743

QC 20260716

Available from: 2026-07-16 Created: 2026-07-16 Last updated: 2026-07-16Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

ScopusPresentation

Authority records

Tinke, Pjotrvon Mentlen, Thomas

Search in DiVA

By author/editor
Tinke, Pjotrvon Mentlen, Thomas
By organisation
School of Electrical Engineering and Computer Science (EECS)
Information Systems

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 1 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf