Magic: The Gathering as a benchmark for reasoning and strategic intelligence in large language models

Abstract

Current benchmarks for evaluating Large Language Models (LLMs) face critical challenges including saturation, contamination, and limited assessment of complex reasoning capabilities. We propose Magic: The Gathering (MTG) as a novel benchmark that addresses these limitations through its proven Turing-completeness, massive state space complexity, and unique combination of rule-based reasoning, strategic planning, and natural language understanding requirements. This position paper presents a comprehensive framework for MTG-based evaluation, including puzzle hierarchies, multi-dimensional metrics, and scalable difficulty progression from simple rule verification to complex multi-turn strategic scenarios. Our analysis demonstrates how MTG’s computational intractability and constantly evolving card pool create a contamination-resistant benchmark that can differentiate between pattern matching and genuine reasoning capabilities, providing crucial insights for AGI development.

1. Introduction and motivation

The rapid advancement of Large Language Models has exposed fundamental limitations in current evaluation methodologies. Leading models now achieve 80-90% accuracy on traditional benchmarks like MMLU, GSM8K, and HumanEval, creating a saturation crisis that obscures meaningful capability differences. More critically, contamination affects 42% of papers using popular benchmarks, with models potentially memorizing test data rather than demonstrating genuine reasoning¹,²,³. The emergence of game-based evaluation frameworks from Deep Blue’s chess victory to CICERO’s Diplomacy mastery has proven invaluable for measuring AI capabilities in controlled yet complex environments⁴. However, these benchmarks typically focus on specific cognitive domains rather than testing the integrated reasoning required for artificial general intelligence.

Magic: The Gathering presents an extraordinary opportunity to advance LLM evaluation. With over 20,000 unique cards, a 200+ page comprehensive rulebook, and proven Turing-completeness⁵,⁶, MTG creates a computational environment where optimal play is undecidable harder than the Halting Problem itself. Unlike chess with its 10⁴⁷ possible positions or even Go with 10¹⁷², MTG’s state space approaches 10⁷⁰⁰ in drafting alone, with game trajectories spanning 10⁴⁰ possibilities⁷. This complexity, combined with imperfect information, resource management across five color systems, and a stack-based priority system for spell resolution, creates evaluation scenarios that resist both memorization and brute-force approaches.

The game’s continuous evolution releasing 200-300 new cards quarterly with novel mechanics provides natural contamination resistance while ensuring benchmark longevity. Furthermore, MTG uniquely combines multiple cognitive challenges: parsing complex natural language rules, maintaining game state consistency, strategic planning under uncertainty, probabilistic reasoning about hidden information, and resource optimization across multiple constraints. This positions MTG as an ideal testbed for evaluating progress toward AGI, where success requires not just pattern recognition but genuine reasoning, planning, and adaptability.

2. Current landscape of AI evaluation

Traditional benchmark limitations reveal fundamental gaps

The contemporary LLM evaluation ecosystem demonstrates both remarkable progress and critical limitations. Big-Bench Hard, designed to challenge models with tasks where prior LLMs failed to match human performance, has seen its 23 challenging tasks largely conquered through chain-of-thought prompting⁸,⁹. The recent Big-Bench Extra Hard attempts to address this with tasks achieving only 9.8% harmonic average accuracy for general models, yet this arms race of difficulty escalation treats symptoms rather than underlying evaluation deficiencies¹⁰.

HELM’s comprehensive framework evaluates models across 42 scenarios measuring accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency¹¹,¹². While this multi-dimensional approach represents progress, the framework’s $10,000+ evaluation cost and focus on isolated capabilities fails to capture integrated reasoning¹³. The emergence of Flash HELM, reducing costs by 99%, demonstrates the field’s recognition of scalability challenges but doesn’t address fundamental measurement limitations¹⁴.

Game-based benchmarks have emerged as promising alternatives. Recent frameworks evaluate strategic reasoning through multiple games, combining abstract strategy, hidden information, and social deduction¹⁵,¹⁶,¹⁷. However, these benchmarks typically evaluate narrow strategic capabilities rather than the comprehensive reasoning required for complex real-world tasks. Current multi-game benchmarks reveal that LLMs struggle with deep multi-hop reasoning, yet lack the computational complexity to truly differentiate advanced reasoning from sophisticated pattern matching¹⁸.

Multi-capability benchmarks expose performance gaps

Recent multi-modal and composite benchmarks reveal striking capability gaps when AI systems face integrated challenges. GAIA’s seemingly simple tasks achieve 92% human success versus 15% for GPT-4 with plugins, demonstrating how multi-step reasoning with tool use remains fundamentally challenging¹⁹. SWE-bench’s software engineering tasks see the best models solving only 20% of real GitHub issues, despite these systems excelling at isolated coding challenges²⁰,²¹. WebArena’s 14.41% AI success rate versus 78.24% human performance on web navigation tasks further illustrates the chasm between current capabilities and human-level general intelligence²².

The contamination crisis compounds these challenges. LiveBench’s monthly updates using recent arXiv papers, news, and competitions represents one solution, achieving below 70% accuracy for top models while resisting memorization²³. However, this approach requires constant human effort and doesn’t address the need for benchmarks that inherently resist contamination through computational complexity rather than temporal novelty²⁴,²⁵.

3. Games as AI benchmarks: lessons from history

Evolution from perfect to imperfect information challenges

The progression of game-based AI evaluation illuminates both achievements and limitations in measuring machine intelligence. Chess, with its 10⁴⁷ possible positions, provided early insights into search algorithms and evaluation functions²⁶. Deep Blue’s 1997 victory demonstrated brute-force computation could overcome human intuition in perfect information games. AlphaZero’s 2017 self-play mastery revealed how neural networks could learn strategic concepts from scratch, yet chess’s deterministic nature limits its relevance to real-world uncertainty²⁷,²⁸.

Go’s 10¹⁷² state space challenged the brute-force paradigm, requiring pattern recognition and intuitive evaluation that seemed uniquely human. AlphaGo’s “Move 37” against Lee Sedol demonstrated creative play transcending human strategic understanding. However, Go remains a perfect information game where all relevant data is visible, unlike most real-world decision scenarios²⁹,³⁰.

The transition to imperfect information games marked a crucial evolution. Libratus and Pluribus conquering poker demonstrated AI’s ability to reason under uncertainty, model opponents, and execute strategic deception. Poker’s hidden information and probabilistic outcomes better approximate real-world challenges, yet its relatively constrained action space and mathematical framework limit broader applicability³¹,³²,³³.

CICERO’s achievement in Diplomacy represents the current frontier combining strategic reasoning with natural language negotiation across seven-player games. This integration of communication and strategy approaches the complexity of human interaction, yet Diplomacy’s structured rules and zero-sum territorial dynamics still simplify real-world negotiation and cooperation³⁴,³⁵,³⁶. Each game benchmark has advanced our understanding while revealing new limitations, suggesting the need for evaluation environments matching the full complexity of human reasoning.

Computational complexity as a discriminator

Game complexity metrics provide crucial insights for benchmark design. State-space complexity indicates computational requirements for exhaustive analysis, while game-tree complexity measures strategic depth through possible game variations³⁷. Traditional benchmarks cluster at manageable complexity levels: chess at 10¹²³ possible games remains theoretically solvable given sufficient computation, while poker’s mathematical framework enables Nash equilibrium approximation.

Magic: The Gathering exists in a fundamentally different complexity class. Its Turing-completeness means optimal play is formally undecidable not merely computationally intractable but theoretically impossible to solve³⁸,³⁹. The 2019 proof by Churchill, Biderman, and Herrick demonstrated MTG can encode arbitrary Turing machines in legal gameplay, making it at least as hard as the Halting Problem. Even verifying whether a single move is legal proves coNP-complete, requiring exponential time in worst cases⁴⁰.

This extreme complexity serves as a natural discriminator between genuine reasoning and sophisticated pattern matching. While models might memorize common chess openings or poker strategies, MTG’s vast state space and complex interactions make memorization infeasible. Success requires understanding abstract principles, reasoning about novel combinations, and adapting strategies to unprecedented situations capabilities essential for artificial general intelligence.

4. Magic: The Gathering’s unique evaluation properties

Computational intractability ensures longevity

MTG’s proven Turing-completeness fundamentally distinguishes it from all existing game benchmarks. The ability to encode arbitrary computations means no algorithm can determine optimal play in general cases a theoretical barrier that ensures perpetual challenge regardless of computational advances⁴¹,⁴². Unlike benchmarks that become obsolete through saturation, MTG’s undecidability creates an evaluation ceiling that cannot be reached through incremental improvements in model scale or training data.

The game’s 20,000+ unique cards create combinatorial explosion in deck construction alone, with legal Standard format decks producing approximately 10⁶⁰ possible configurations. Each card introduces distinct mechanics, triggered abilities, and interaction patterns that must be parsed from natural language text averaging 50-100 words. The comprehensive rules document exceeds 200 pages, implementing a seven-layer interaction system for resolving conflicting effects, a stack-based priority system for spell timing, and state-based actions that continuously monitor game conditions⁴³.

This complexity manifests in practical gameplay through multiple simultaneous decision surfaces. Players must optimize resource allocation across five color systems, each with distinct strategic philosophies and mechanical themes. The mana system creates tension between powerful late-game cards and consistent early execution. Life totals function as expendable resources, creating strategic decisions about damage races versus defensive positioning. Card advantage calculations must account for virtual advantage through threat diversity and tempo gains through efficient mana usage.

Multi-modal reasoning requirements create comprehensive evaluation

MTG uniquely exercises diverse cognitive capabilities within a single evaluation framework. Natural language understanding challenges arise from parsing card text that combines templates, keywords, and novel abilities⁴⁴. Cards reference complex timing windows (“at the beginning of your upkeep”), conditional triggers (“whenever a creature you control dies”), and layered effects (“creatures you control get +1/+1 and have vigilance”). Models must disambiguate pronouns, resolve syntactic ambiguity, and maintain semantic consistency across thousands of distinct effects.

Strategic planning operates across multiple temporal horizons. Immediate tactical decisions which creature blocks, what spell to counter interact with medium-term positional goals and long-term win condition assembly. Hidden information from opponents’ hands and library contents requires probabilistic reasoning and Bayesian updating as games progress. The metagame layer adds another dimension: successful play requires understanding format dynamics, predicting opponent strategies, and selecting appropriate counter-strategies.

Rule consistency and state management present unprecedented challenges for current AI systems. The stack system requires tracking multiple spell resolutions in last-in-first-out order while maintaining priority passes between players. Continuous effects, triggered abilities, and state-based actions create complex interdependencies where changing one game element cascades through multiple rules interactions. Replacement effects modify how events occur, creating nested conditionals that challenge logical reasoning capabilities.

5. Proposed MTG-LLM evaluation framework

Hierarchical puzzle design enables progressive evaluation

Our framework structures MTG evaluation through carefully designed puzzle hierarchies that isolate and combine different reasoning capabilities. Level 1: Rule Comprehension presents single cards or simple interactions, asking models to explain effects, identify legal targets, or determine trigger conditions. These puzzles test natural language understanding and basic rules knowledge without strategic complexity. Example: “You control a 2/2 creature with flying. Your opponent controls a 3/3 creature without flying and a 1/1 creature with reach. Which creatures can block your flying creature?”

Level 2: State Evaluation introduces board positions requiring multi-step reasoning about game rules. Models must track multiple permanents, resolve triggered abilities in correct order, and determine legal action sequences. These puzzles test working memory, logical consistency, and understanding of rules interactions. Example: “You control three Mountains and a Goblin Electromancer (creature spells you cast cost 1 less). You have Lightning Bolt and Shock in hand. Your opponent is at 4 life with no creatures. Describe all possible sequences of plays and their outcomes.”

Level 3: Strategic Decision-Making presents complex game states requiring evaluation of multiple strategic lines. Models must consider tempo versus card advantage, risk assessment under uncertainty, and resource optimization across multiple turns. Success requires not just finding legal plays but identifying optimal strategies given incomplete information.

Level 4: Deck Construction and Metagame Reasoning challenges models to build coherent decks given format constraints, evaluate deck matchups, and predict metagame evolution. This level tests abstract reasoning about strategic systems, understanding of statistical distributions, and ability to balance competing objectives. A model that performs well here demonstrates understanding of high-level strategic concepts beyond memorized patterns.

Multi-dimensional metrics capture reasoning quality

Traditional accuracy metrics insufficiently capture reasoning capabilities in complex domains. Our framework implements multiple evaluation dimensions to provide comprehensive assessment⁴⁵. Rules Adherence Score quantifies legal move generation accuracy, tracking both false positives (illegal moves proposed) and false negatives (legal moves missed). This metric uses automated rules engines for verification, ensuring objective and reproducible evaluation without human judgment variability.

Strategic Optimality Metrics evaluate decision quality through multiple approaches. Win probability assessment uses Monte Carlo simulations from decision points to measure expected value of chosen actions. Comparative analysis benchmarks against human expert decisions and existing AI systems. Explanation quality scoring evaluates reasoning transparency: can models articulate why certain plays are superior? This combines automated coherence checking with structured rubrics for strategic reasoning evaluation.

Generalization Assessment tests transfer learning and adaptation capabilities. Novel card evaluation introduces previously unseen cards with new mechanics, measuring whether models can reason from first principles rather than memorized patterns. Format adaptation challenges models to adjust strategies when card pools change, testing meta-learning capabilities. Variant rule handling introduces modified game rules, assessing whether models understand underlying principles or merely memorize specific configurations.

Computational Efficiency Tracking measures resource requirements for different evaluation levels. Time-to-decision metrics capture reasoning speed, important for practical applications. Memory usage patterns reveal whether models maintain appropriate game state representations. Scaling behavior analysis examines how performance changes with increased computational resources, distinguishing models that benefit from deeper search versus those hitting fundamental reasoning limitations.

Ensuring reproducibility through deterministic scenarios

Reproducibility challenges plague current LLM evaluation, with stochastic generation and model updates creating inconsistent results⁴⁶. Our framework addresses these through carefully designed deterministic evaluation protocols. Fixed puzzle sets with canonical solutions enable consistent comparison across models and versions. Each puzzle includes multiple valid solution paths, but scoring remains deterministic through standardized evaluation functions.

Version control systems track puzzle evolution, model configurations, and evaluation results. Puzzle databases include creation date stamps, difficulty ratings validated through human testing, and detailed solution explanations. This enables longitudinal studies of model improvement and identification of capability regressions from updates.

Contamination resistance emerges naturally from MTG’s complexity and evolution. With new card sets released quarterly, evaluation sets can incorporate recent cards unlikely to appear in training data⁴⁷. Procedural puzzle generation using constraint satisfaction creates unlimited evaluation scenarios while maintaining difficulty control. Community validation through player testing ensures puzzles represent realistic game scenarios rather than artificial constructions.

6. Theoretical advantages over existing benchmarks

Natural contamination resistance through complexity and evolution

MTG’s quarterly release cycle introduces 200-300 new cards with novel mechanics, creating natural temporal boundaries for contamination-free evaluation. Unlike static benchmarks requiring manual updates, MTG’s evolution is inherent to the game’s design. New mechanics fundamentally alter strategic landscapes: the introduction of Planeswalker cards created new resource axes, while modal double-faced cards challenged traditional card advantage calculations. Models trained before these innovations cannot rely on memorized patterns, requiring genuine reasoning to handle novel interactions.

The sheer combinatorial space makes comprehensive training data coverage impossible. With 10⁶⁰ possible Standard decks and exponentially more possible game states, no training dataset can encompass even a fraction of possible scenarios⁴⁸. Each game generates unique positions through shuffled libraries, mulligan decisions, and player choices. This differs fundamentally from chess, where opening databases cover common positions, or poker, where optimal strategies for simplified variants are mathematically solved.

Procedural puzzle generation amplifies contamination resistance. By programmatically combining cards and board states, we can generate unlimited evaluation scenarios that have never existed in human play. Constraint satisfaction ensures puzzles remain solvable and realistic while guaranteeing uniqueness. This approach scales infinitely without human annotation effort, addressing the resource limitations that constrain benchmarks like LiveBench’s monthly updates.

Genuine reasoning discrimination through undecidability

MTG’s Turing-completeness creates qualitative differences from existing benchmarks. In chess or Go, perfect play is theoretically computable given sufficient resources limitations are merely computational. In MTG, optimal play is formally undecidable, creating situations where no amount of computation or training data enables perfect performance⁴⁹,⁵⁰. This theoretical ceiling ensures perpetual challenge regardless of AI advances.

The undecidability manifests in practical evaluation through specific puzzle types. Self-referential card interactions create logical paradoxes requiring careful rules parsing to resolve. Infinite loops must be detected and handled according to tournament rules. Some board states have indeterminate outcomes depending on player choices in mandatory loops. These scenarios test whether models truly understand logical systems or merely pattern-match against training examples.

Discrimination between reasoning and memorization becomes clearer in complex scenarios. When faced with novel card combinations, models must derive interactions from first principles rather than retrieving memorized outcomes. The seven-layer interaction system for resolving effects requires maintaining logical consistency across multiple rule applications. Success requires building mental models of game state evolution, not just matching patterns to previous examples.

7. Implementation considerations and challenges

Computational requirements and optimization strategies

Full MTG gameplay simulation presents significant computational challenges. Complete rules engines must track hundreds of game objects, continuous effects, and triggered abilities simultaneously. State space representation for complex board positions can require megabytes of memory. Monte Carlo tree search for strategic evaluation generates millions of simulated games, each requiring rules engine execution.

Our framework addresses these challenges through hierarchical evaluation design. Early puzzle levels require minimal computation, testing comprehension rather than search. Focused scenarios reduce state space by presenting specific game situations rather than complete matches. This enables meaningful evaluation without prohibitive computational costs, making the benchmark accessible to academic researchers without massive compute budgets.

Optimization strategies leverage existing infrastructure where possible. Open-source rules engines provide verified implementations for game mechanics validation⁵¹. These engines handle the complex rules interactions, allowing evaluation to focus on decision-making rather than rules implementation. API layers abstract rules complexity, presenting cleaner interfaces for LLM interaction while maintaining full game fidelity.

Creating comprehensive puzzle datasets

Initial dataset creation requires collaboration between AI researchers and MTG experts. Professional players and judges provide strategic puzzles that test genuine understanding rather than rules trivia. Coverage analysis ensures puzzles span different strategic archetypes: aggressive, control, combo, and midrange strategies each require distinct reasoning patterns.

Difficulty calibration uses human testing to establish baseline performance. Puzzles should span from trivial (testing basic comprehension) to expert-level (challenging professional players). Systematic difficulty progression enables meaningful capability measurement across model scales. Human performance baselines provide context for AI achievement levels.

Long-term dataset growth leverages community contributions and procedural generation. A puzzle submission platform enables community creation with validation workflows. Automated puzzle generation using constraint programming creates unlimited scenarios while maintaining quality. Statistical analysis of puzzle characteristics ensures balanced coverage across evaluation dimensions.

8. Future research directions

Integration with multi-agent evaluation

MTG’s multiplayer formats offer unique opportunities for multi-agent AI evaluation. Commander format’s four-player games with political alliances and betrayals test social reasoning beyond two-player zero-sum scenarios. Teams formats require coordination between AI agents sharing hidden information. These scenarios evaluate capabilities essential for real-world AI deployment where cooperation and competition coexist.

Research directions include tournament structures where AI agents compete over multiple games, requiring metagame adaptation and opponent modeling. Mixed human-AI play evaluates whether models can effectively collaborate with human partners. Communication protocols testing whether models can explain strategies and negotiate table dynamics advance natural language capabilities beyond simple dialogue.

Curriculum learning and capability progression

The hierarchical puzzle structure enables curriculum learning approaches where models progressively acquire capabilities. Foundation skills in rules comprehension support advanced strategic reasoning. Transfer learning from simple to complex formats tests generalization. This progression mirrors human learning, where players advance from basic rules understanding to tournament-level play.

Developmental evaluation tracks capability acquisition over training, identifying phase transitions in understanding. Error analysis reveals whether failures stem from rules misunderstanding, strategic miscalculation, or reasoning limitations. These insights inform architecture improvements and training methodologies for more capable systems.

Bridging to real-world reasoning tasks

MTG’s complexity characteristics natural language parsing, state management, strategic planning, and uncertainty reasoning mirror challenges in real-world domains. Legal reasoning requires similar rules interpretation and precedence handling. Medical diagnosis involves comparable uncertainty management and multi-factor optimization. Software engineering demands equivalent state tracking and constraint satisfaction.

Research validating correlations between MTG performance and real-world reasoning capabilities could establish the benchmark’s broader significance. Transfer learning experiments from MTG-trained models to professional domains test whether the game develops generalizable reasoning. These connections position MTG evaluation as more than game-playing assessment, but as a probe for artificial general intelligence.

9. Conclusion

Magic: The Gathering represents a fundamental advance in LLM evaluation methodology, addressing critical limitations of current benchmarks through its unique combination of computational complexity, natural language reasoning, and strategic depth. Its proven Turing-completeness ensures perpetual challenge, while continuous evolution provides natural contamination resistance. The game’s multi-modal cognitive requirements from parsing complex card text to managing hidden information create comprehensive evaluation scenarios that discriminate genuine reasoning from sophisticated pattern matching.

Our proposed framework provides practical implementation paths through hierarchical puzzle design, multi-dimensional evaluation metrics, and reproducibility protocols. By structuring evaluation from simple rules comprehension to complex strategic reasoning, the framework accommodates models across capability levels while maintaining meaningful discrimination at the frontier. Integration with existing rules engines and community resources makes implementation feasible without prohibitive development costs.

The implications extend beyond game-playing evaluation. MTG’s reasoning requirements mirror real-world challenges in legal interpretation, medical diagnosis, and strategic planning. Success in this domain suggests capabilities applicable to professional contexts where natural language understanding, logical consistency, and decision-making under uncertainty converge. As the field progresses toward artificial general intelligence, benchmarks must evolve beyond saturated academic tests to challenges that truly probe reasoning capabilities.

We call for the research community to collaborate in developing this benchmark, contributing puzzles, refining evaluation metrics, and establishing human performance baselines. Through collective effort, MTG-based evaluation can provide crucial insights into AI reasoning capabilities, identifying both achievements and limitations as we advance toward more capable and reliable AI systems. The undecidable complexity of optimal MTG play ensures this benchmark will remain relevant regardless of future computational advances, providing a north star for AGI development.

References

1. Golchin, S., & Surdeanu, M. (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. ACL Anthology. https://aclanthology.org/2023.findings-emnlp.722/
2. Wei, J., et al. (2024). Evaluation data contamination in LLMs: how do we measure it and (when) does it matter? arXiv preprint arXiv:2411.03923.
3. Yang, S., et al. (2024). Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. arXiv preprint arXiv:2402.15938.
4. Bakhtin, A., et al. (2022). Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067-1074.
5. Churchill, A., Biderman, S., & Herrick, A. (2019). Magic: The Gathering is Turing Complete. arXiv preprint arXiv:1904.09828.
6. Clark, A. (2019). Magic: The Gathering is officially the world’s most complex game. MIT Technology Review. https://www.technologyreview.com/2019/05/07/135482/magic-the-gathering-is-officially-the-worlds-most-complex-game/
7. Ward, D., et al. (2020). AI solutions for drafting in Magic: the Gathering. arXiv preprint arXiv:2009.00655.
8. Suzgun, M., et al. (2022). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. GitHub. https://github.com/suzgunmirac/BIG-Bench-Hard
9. Confident AI. (2024). Top LLM Benchmarks Explained: MMLU, HellaSwag, BBH, and Beyond. https://www.confident-ai.com/blog/llm-benchmarks-mmlu-hellaswag-and-beyond
10. Metz, L., et al. (2024). BIG-Bench Extra Hard. arXiv preprint arXiv:2502.19187.
11. Liang, P., et al. (2022). Holistic Evaluation of Language Models. arXiv preprint arXiv:2211.09110.
12. OpenReview. (2023). Holistic Evaluation of Language Models. https://openreview.net/forum?id=iO4LZibEqW
13. Hugging Face. (2023). The Big Benchmarks Collection. https://huggingface.co/collections/open-llm-leaderboard/the-big-benchmarks-collection-64faca6335a7fc7d4ffe974a
14. IBM Research. (2024). Evaluate your LLM using 99% less compute. https://research.ibm.com/blog/efficient-llm-benchmarking
15. PlayBench. (2024). AI Models Performance. https://playbench.ai/models
16. Google. (2024). Kaggle Game Arena evaluates AI models through games. https://blog.google/technology/ai/kaggle-game-arena/
17. Yue, X., et al. (2024). Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play. arXiv preprint arXiv:2508.03368.
18. Tian, Y., et al. (2024). SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially? arXiv preprint arXiv:2503.12349.
19. Mialon, G., et al. (2023). GAIA: a benchmark for General AI Assistants. arXiv preprint arXiv:2311.12983.
20. Jimenez, C., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? GitHub. https://github.com/SWE-bench/SWE-bench
21. Cognition. (2024). SWE-bench technical report. https://cognition.ai/blog/swe-bench-technical-report
22. Zhou, S., et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv preprint arXiv:2307.13854.
23. White, J., et al. (2024). LiveBench: A Challenging, Contamination-Limited LLM Benchmark. arXiv preprint arXiv:2406.19314.
24. Reiter, E. (2024). I’m very worried about data contamination. Ehud Reiter’s Blog. https://ehudreiter.com/2024/03/12/data-contamination-worries/
25. Ding, N., et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv preprint arXiv:2406.04244.
26. Chess.com. (2024). From Deep Blue to AlphaZero: The Rise of AI in Contemporary Chess. https://www.chess.com/blog/sourabhjoshi01/from-deep-blue-to-alphazero-the-rise-of-ai-in-contemporary-chess
27. Wikipedia. (2024). AlphaZero. https://en.wikipedia.org/wiki/AlphaZero
28. Chess.com. (2024). AlphaZero - Chess Engines. https://www.chess.com/terms/alphazero-chess-engine
29. Wikipedia. (2024). AlphaGo. https://en.wikipedia.org/wiki/AlphaGo
30. Google DeepMind. (2024). AlphaGo. https://deepmind.google/research/projects/alphago/
31. Wikipedia. (2024). Libratus. https://en.wikipedia.org/wiki/Libratus
32. Poker.org. (2024). This was the week… AI took on the poker pros and won. https://www.poker.org/latest-news/this-was-the-week…-ai-took-on-the-poker-pros-and-won-aFeFL7y8OWw9/
33. Science. (2019). Superhuman AI for multiplayer poker. https://www.science.org/doi/10.1126/science.aay2400
34. Science. (2022). Human-level play in the game of Diplomacy by combining language models with strategic reasoning. https://www.science.org/doi/10.1126/science.ade9097
35. Meta AI. (2022). CICERO. https://ai.meta.com/research/cicero/
36. The Conversation. (2022). An AI named Cicero can beat humans in Diplomacy, a complex alliance-building game. https://theconversation.com/an-ai-named-cicero-can-beat-humans-in-diplomacy-a-complex-alliance-building-game-heres-why-thats-a-big-deal-195208
37. Wikipedia. (2024). Game complexity. https://en.wikipedia.org/wiki/Game_complexity
38. Churchill, A., Biderman, S., & Herrick, A. (2019). Magic: The Gathering is Turing Complete. arXiv preprint arXiv:1904.09828.
39. MIT Technology Review. (2019). Magic: The Gathering is officially the world’s most complex game. https://www.technologyreview.com/2019/05/07/135482/magic-the-gathering-is-officially-the-worlds-most-complex-game/
40. Chatterjee, K., & Ibsen-Jensen, R. (2016). The Complexity of Deciding Legality of a Single Step of Magic: The Gathering. Semantic Scholar. https://www.semanticscholar.org/paper/The-Complexity-of-Deciding-Legality-of-a-Single-of-Chatterjee-Ibsen-Jensen/f344e10296c76366b70b367b5a7ded9c3d28cca7
41. arXiv. (2019). Magic: The Gathering is Turing Complete. https://arxiv.org/abs/1904.09828
42. ResearchGate. (2019). Magic: The Gathering is Turing Complete. https://www.researchgate.net/publication/332590574_Magic_The_Gathering_is_Turing_Complete
43. Academic Kids. (2024). Game complexity. https://academickids.com/encyclopedia/index.php/Game-tree_complexity
44. Cao, S., et al. (2024). A Taxonomy of Collectible Card Games from a Game-Playing AI Perspective. arXiv preprint arXiv:2410.06299.
45. Orq AI. (2024). LLM Benchmarks Explained: Significance, Metrics & Challenges. https://orq.ai/blog/llm-benchmarks
46. AIMultiple. (2024). Reproducible AI: Why it Matters & How to Improve it. https://research.aimultiple.com/reproducible-ai/
47. LiveBench. (2024). A Challenging, Contamination-Free LLM Benchmark. https://github.com/LiveBench/LiveBench
48. Ward, D., et al. (2020). AI solutions for drafting in Magic: the Gathering. arXiv preprint arXiv:2009.00655.
49. Churchill, A., Biderman, S., & Herrick, A. (2019). Magic: The Gathering is Turing Complete. https://ar5iv.labs.arxiv.org/html/1904.09828
50. Gradient Science. (2024). Do Large Language Model Benchmarks Test Reliability? https://gradientscience.org/platinum-benchmarks/
51. Wingedsheep. (2024). Mana from the Machine: An AI-Powered MTG Set Generator. https://wingedsheep.com/mana-from-the-machine/