Ai Machine Learning · arXiv:2610.13948
Can Language Models Play the Wikipedia Game?
Authors: Alexander Hickey, Joanna Knight, James Bishop (arXiv:2610.13948) · arXiv: 2610.13948
Core Methodological Innovation
Featured in Nujan's Ai Machine Learning Literature Digest. Open side-by-side with interactive equation, methodology, and citation breakdowns.
Key Quantitative & Theoretical Takeaway: Open arXiv:2610.13948 in Nujan to inspect every mathematical symbol, experimental baseline, and citation directly alongside the original PDF.
Abstract
Language models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. We used the Wikipedia Game to test language models on an analogous information retrieval task. In the game, players must reach a target article using only article links. We compared sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets. We construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex and provide code to build and navigate them. This allows the study of language models' navigation capabilities without relying on third-party APIs. We compared the models with human and TF-IDF benchmarks. The Wikipedia Game format enables robust comparisons across techniques, languages, and domains. No model significantly outperformed humans, although the best models (GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B) achieved comparable success. Performance and rankings varied across languages and topics, with implications for practitioners utilising these techniques.
Step-by-Step Equation & Methodology Breakdown
What is Can Language Models Play the Wikipedia Game? (arXiv:2610.13948) about?
Language models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. We used the Wikipedia Game to test language models on an analogous information retrieval task. In the game, players must reach a target article using only article links. We compared sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets. We construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex and provide code to build and navigate them. This allows the study of language models' navigation capabilities without relying on third-party APIs. We compared the models with human and TF-IDF benchmarks. The Wikipedia Game format enables robust comparisons across techniques, languages, and domains. No model significantly outperformed humans, although the best models (GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B) achieved comparable success. Performance and rankings varied across languages and topics, with implications for practitioners utilising these techniques.
How can I read arXiv:2610.13948 with interactive explanations?
Visit https://nujan.app/abs/2610.13948 to open Can Language Models Play the Wikipedia Game? in Nujan's split-screen research workspace with instant equation and methodology breakdowns.