When did this chess game against ChatGPT diverge from theory?

Question

I have recently been tinkering on and off with ways to make ChatGPT play chess. This has worked somewhat well, even with ChatGPT3.5, using a prompt crafted to make ChatGPT re-prompt itself at every turn with the full game history so far. I would estimate that it played at maybe 1000-1100 elo level, whereas GPT-4 reached maybe 1400. By computer chess standards that is obviously abysmal, but I'm fascinated with it as this thing is not meant to play chess.

My most recent attempt with this little pastime involved augmenting my usual prompt with a single, chosen to be instructive example of good play. Concretely, I gave it Morphy's Opera Game and asked it to emulate White's playstyle. The exact prompt and game can be found here (i.e. chapter 15 in the study).

In my previous experiments, coherence of play was mostly lost when the sum total of all prompting and self-prompting during a game ongoing in this manner reached the size of the context window (but this could be remedied by reprompting with only the game history, i.e. removing chat history). Without having counted tokens, I expect that this was the same here, with time to decoherence shortened slightly due to the Morphy game prompt.

However, up to that point, I was totally blown away by the quality of ChatGPT's play in this game.

[FEN ""]
[White "ChatGPT 3.5"]
[Black "Shredder Chess Online (level hard)"]
[Result "0-1"]
[UTCDate "2023.04.13"]
[Variant "Standard"]
[ECO "C45"]
[Opening "Scotch Game: Modern Defense"]

1. e4 { [%eval 0.36] } { Prompt was:
We are playing a chess game. At each turn, answer in a code box strictly in the following format:
PGN of the game so far:
Best move for White:
---
Here is an example game where White played well:
(...followed by a commented PGN of the famous Opera Game by Morphy...)
Try to play like White did in that game. } 1... e5 { [%eval 0.35] } 2. Nf3 { [%eval 0.29] } 2... Nc6 { [%eval 0.32] } 3. d4 { [%eval 0.13] } 3... exd4 { [%eval 0.13] } 4. Nxd4 { [%eval 0.03] } 4... Qh4 { [%eval 0.57] } 5. Nc3 { [%eval 0.95] } 5... Bb4 { [%eval 0.87] } 6. Be2 { [%eval 0.84] } 6... Bxc3+ { [%eval 0.93] } 7. bxc3 { [%eval 0.94] } 7... Nf6? { [%eval 2.22] } { Mistake. Nxd4 was best. } (7... Nxd4 8. Qxd4 Ne7 9. c4 O-O 10. Bb2 Qh6 11. f4 d6 12. O-O) 8. O-O? { [%eval 0.55] } { Mistake. Nf5 was best. } (8. Nf5 Qxe4 9. Nxg7+ Kd8 10. O-O Qg6 11. Nh5 Rg8 12. Ng3 d6) 8... O-O?! { [%eval 1.14] } { Inaccuracy. Nxd4 was best. } (8... Nxd4) 9. Nf5 { [%eval 0.85] } 9... Qxe4 { [%eval 1.28] } 10. Bd3 { [%eval 1.11] } 10... Qe5 { [%eval 1.18] } 11. Re1 { [%eval 1.49] } 11... Qxc3? { [%eval 3.22] } { Mistake. Qa5 was best. } (11... Qa5 12. Rb1) 12. Bd2?! { [%eval 2.42] } { leandroac - FreiwiedieGeier, 0-1, -, https://lichess.org/yD80NGqu } (12. Rb1 d6 13. Bb2 Qc5 14. Nh6+ Kh8 15. Bxf6 gxf6 16. Rb5 Qc3 17. Rh5 Ne5 18. Bxh7 Re8) 12... Qb2? { [%eval 4.49] } { Mistake. Qc5 was best. } (12... Qc5) 13. Rb1 { [%eval 4.61] } 13... Qxa2 { [%eval 5.61] } 14. Bc3 { [%eval 5.39] } 14... d6 { [%eval 4.64] } 15. Bxf6 { [%eval 4.14] } 15... Bxf5 { [%eval 4.1] } 16. Bxf5 { [%eval 4.17] } 16... Qc4?! { [%eval 5.77] } { Inaccuracy. Ne5 was best. } (16... Ne5 17. Bxe5 dxe5 18. Rxb7 g6 19. Bd3 Rfd8 20. Qa1 Qxa1 21. Rxa1 a5 22. Rxc7 a4 23. h3) 17. Re4 { [%eval 5.6] } (17. Qh5 h6 18. Re4 Qa2 19. Rf1 Ne5 20. Rg4 Ng6 21. Bxg7)  (17. Bxh7+ Kxh7 18. Qh5+ Kg8 19. Qg5) 17... Nd4?! { [%eval 9.16] } { Inaccuracy. Qa2 was best. } (17... Qa2 18. Ra1 Qxa1 19. Qxa1 g6 20. Bd7 Ne5 21. Bxe5 Rfe8 22. Bh8 Re5 23. Bxe5 dxe5 24. Qxe5) 18. Rxd4 { [%eval 9.16] } 18... Qa2? { [%eval #9] } { Checkmate is now unavoidable. g6 was best. } (18... g6 19. Rxc4 Rae8 20. Rxc7 Re5 21. Bxe5 dxe5 22. Be4 Kh8 23. Qd5 g5 24. Rcxb7) 19. Bxh7+ { [%eval #7] } 19... Kxh7 { [%eval #7] } 20. Rh4+?? { [%eval 6.59] } { Lost forced checkmate sequence. Rg4 was best. } (20. Rg4 Qxb1 21. Rxg7+ Kh8 22. Rg4+ Kh7 23. Qxb1 Rg8 24. c3+ Rg6 25. Rh4+ Kg8 26. Rh8#) 20... Kg6 { [%eval 6.69] } 21. Qh5+?! { [%eval 5.1] } { Inaccuracy. Bc3 was best. } (21. Bc3 Qxb1 22. Qxb1 f5 23. Qb5 Rae8 24. Qd7 Re5 25. Rh3 f4 26. Rh4 b5 27. Qg4+ Rg5) 21... Kxf6 { [%eval 5.29] } 22. Re1 { [%eval 5.16] } 22... g6 { [%eval 4.72] } 23. Rf4+ Kg7 { Here, the game was adjourned, because ChatGPT3.5 lost track of the position. } 0-1

This of course begs the question whether the game could have been in its training data. The lichess database suggests that the game departed from other games in the database when Shredder Chess Online played 12. ... Qb2.

Are there any other games anywhere that follow this game for longer, and if so, what is the source?

Answer 1

There's not always a clear-cut point at which a game deviates from theory. A common use of the term refers to the first move that hasn't been played before, and as 10...Qe5 is the first move that doesn't appear on Lichess masters' database, we could say that move is a "novelty" according to the reference database we're using. Most serious chess players would agree with this assessment.

If we consider all games ever played on Lichess, then 12...Qb2 is indeed the "novelty", but most people wouldn't consider that every online Blitz game is a novelty. However for this specific use case it may be more useful to consider "theory" ends here, as there's no reason to expect ChatGPT's training set to prefer master games over amateur games.

Generative AI is basically glorified autocomplete so it's perfectly possible it was aware of a chess game starting with that line, but that would beg the question of why it chose that specific line instead of some other more common variation.