11

Chess engines are so strong that it is becoming increasingly hard to find positions that they are demonstrably evaluating incorrectly. I'm interested in collecting examples—if there are any—of positions where we know the correct evaluation, but that are very difficult for engines.

Below is a beautiful study that I encountered recently in Chapter 5 of the book Barrycades and Septoku: Papers in Honor of Martin Gardner and Tom Rodgers, by Thane Plambeck and Tomas Rokicki (eds.), MAA Press, 2020.

[Title "Carlos Pereira dos Santos, White To Play And Win"]
[FEN "bR2r3/2nK4/kp1pp1pp/p1R2p2/1P1P1P1P/p7/P2P3P/8 w - - 0 1"]

The engines I have access to are old and out of date, but after 1.b5+ Ka7 they want to play 2.Rxe8? and miss the winning continuation 2.Rxa8! Nxa8 3.Rc8! Rxc8 4.Kxc8 with a win for White. I would be curious to know if stronger engines are able to find this win. [EDIT: Thanks to Oscar Smith for pointing out that I can download Stockfish 11 for free. It had some difficulty at first, but after searching about 2.5 billion nodes it finally found 2.Rxa8!]

Years ago, there was a ChessBase article on this topic. They give some examples of positions, such as fortresses, where the engines' numerical evaluation is way off. However, sometimes the engine plays the position correctly anyway, finding the drawing moves even though it thinks it's losing, and such examples are not so interesting to me. What is more interesting to me is a position where the engine loses a position that a perfect player would draw, or draws (or even loses) a position that a perfect player would win.

Probably the best example from that article is a study by Behting, shown below.

[Title "K. K. Behting, Baltische Schachblätter 1908, White To Play And Draw"]
[FEN "8/8/7p/3KNN1k/2p4p/8/3P2p1/8 w - - 0 1"]

As of 2012, careful analysis demonstrated that the study is sound but that engines had serious trouble with it. Is that still true today? [EDIT: It seems so; after searching 25 billion nodes, Stockfish 11 was still unsuccessfully trying the 1.Ng7+ line that is analyzed in detail here. EDIT 2: See this other chess.SE answer for an update as of 2022.]

There is one other type of example that I know about, namely tablebase positions where White has an extremely subtle win. For example, below is a position from another chess.SE question.

[Title "Lomonosov Tablebase, White To Play And Win"]
[FEN "6N1/3n4/3k1b2/8/1r6/5K1Q/8/8 w - - 3 8"]

Assuming the 50-move rule is not in effect, then 1.Kg2!! wins, while every other move draws (or loses material immediately). But the win takes over 500 moves. An engine without tablebases is unlikely to find 1.Kg2!! and even if it does, sooner or later it is bound to throw away the win.

Another (slightly simpler) example of this sort is the position below, which is a tablebase win for White, but the engines I've tried can't see past the opportunities to draw instantly. But such examples are perhaps not so interesting, since they are so utterly incomprehensible.

[Title "White To Play And Win"]
[FEN "1B6/8/4N3/6n1/8/8/5K2/7k w - - 0 1"]

In the process of typing up this question, I found another question on this site that asks a very similar question. Unfortunately, some of the answers there seem to be (no longer?) valid. Also, let me emphasize again that for my purposes, a fortress position that the engine thinks is losing but that it plays correctly anyway does not count.

In short, my question is this:

Are there other good examples of positions that stump engines?

Improve this question
2
  • 5
    One thing to note is that there is no reason to not have access to top tier new engines. SF and Lc0 are the two best engines, and both are open source and relatively easy to download.
    – Oscar Smith
    Commented May 26, 2020 at 15:57
  • 1
    @OscarSmith : Thanks for the suggestion! I have downloaded Stockfish 11 and have added a couple of comments accordingly.
    – Timothy Chow
    Commented May 27, 2020 at 19:56

5 Answers 5

Reset to default
6

There are plenty of positions that stump engines, and it's not too difficult to find some from engine tournaments. Example, this position stumped Stockfish:

[FEN "1k1r4/pp4pp/4pb2/2Q5/2P5/8/P5PP/6K1 w - - 1 28"]

Stockfish actually reached +2.76 eval later, even though White has no way to break through. In fact Black can afford to give up the e6-pawn and the position is still drawn. Lc0 evaluated the position accurately as a draw, which doesn't mean Lc0 is some kind of goddess at evaluating fortresses, since this position stumped Lc0:

[FEN "5k2/5p2/p3pPp1/3pN2p/b6P/2P5/P5P1/7K w - - 0 35"]

Black plays ...Bb5 and defends along the f1-a6 diagonal, and White will never get anywhere, especially since if the Knight moves too much Black advances the e6-pawn with counterplay. This time Stockfish accurately assessed the position as drawn, while Lc0 was at +1.6 - or about 60% chance to win - or so for quite a while.

If you're wondering about positions with study-like solutions that humans think is pretty - that's a different question.

Improve this answer
2
  • 1
    Thanks for the examples. But if I understand correctly, the engines play these positions correctly even though they misevaluate them, right? That is, the position is actually a draw, and the engine does not throw away the draw by poor play?
    – Timothy Chow
    Commented May 27, 2020 at 4:40
  • 2
    @TimothyChow yes, the engine does not throw away the draw by poor play.
    – Allure
    Commented May 27, 2020 at 4:41
1

Since the definition is "the engine plays the position correctly" (i.e., does not throw away the win or draw), then there are once again many such positions and you can find them from engine tournaments. In these tournaments the opening is randomized, and engines take turns to play White. Sometimes they lose one game and draw the other, therefore losing the opening. Obviously they must have made some mistake in one of the two games, thereby losing/drawing when they could have drawn/won.

Here's an example from the most recent TCEC superfinal, position after 71. Rg4.

[FEN "8/6r1/4kp2/p7/1b4Rp/1P2BP1P/2K3P1/8 b - - 31 71"]

Stockfish (White) had +0.39 evaluation, while Leela (Black) had +0.45 evaluation. Both are well within the range where most games with this eval would be expected to be drawn. Yet Stockfish won, since Leela made what computer chess viewers call "microblunderino" to lose the game.

Of course if the question is "explain in human-understandable terms where Leela's microblunderinos are", then that's a much harder question that takes serious analysis. You can still find some analysis however. Here's a 2021 example (link to game in TCEC archive).

[FEN "5r1k/2nbq1rp/3p2p1/2pPp1P1/p3P3/P3NPQ1/1P2B2R/K6R w - - 3 79"]

Black's last move 78...Kh8 is a serious inaccuracy. There followed 79. Bd3 Bb5? 80. Bc2 Bd7?? 81. Nf5! and Stockfish won with some brilliant tactics that Leela didn't see early enough.

If you're interested in these positions, here's another example, this time with Stockfish on the losing side.

Improve this answer
1

Here's a problem that I learned about from a chess.com YouTube video. According to this web page, in 1995, John Nunn shortened a study by Matous to create this position as a challenge problem for engines. Even today (February 2024), Stockfish 16 on my laptop can't find the winning move 1.Bc7! even after 20 minutes of thinking.

[Title "John Nunn, after Matous, White To Play And Win"]
[FEN "n1QBq1k1/5p1p/5KP1/p7/8/8/8/8 w - - 1 1 "]
Improve this answer
1
  • Interesting. Given 16 cores and 16 GB RAM, SF 16 finds Bc7 in about 3 minutes on my desktop. It spends those 3 minutes analysing checks (pawn captures). When it moves on to analysing bishop moves, it instantly finds that Bc7 is winning.
    – HTTP 410
    Commented Jul 4 at 7:41
1

Here's an interesting example that I saw in the July 2024 issue of The Problemist. I will quote below John Nunn's commentary in full. He encountered this position in a 2023 solving competition.

[Title "Alexey Belenky 4Pr Loshinsky MT 1982, White to play and draw (unsound)"]
[FEN "2N5/2NpnkP1/B4q2/8/rb4K1/6QP/8/8 w - - 1 1 "]

The main line here is 1.Bc4+ d5 2.Sxd5 Qf5+ 3.Kh4 Sg6+ 4.Qxg6+ Qxg6 5.g8Q+ Kxg8 6.Sce7+ Bxe7+ 7.Sf6+ and after Black moves his king there’s a double-pin stalemate. I already had a problem at move two, since after 2...Qxg7+ 3.Kf3 Qxg3+ 4.Kxg3 Ra3+ followed by ...Sxd5 we reach the R+B v B+S balance with opposite-coloured bishops. True, White has an additional pawn at h3, but an extra pawn rarely changes the result unless it is further advanced. I felt that this was very likely to be a win for Black (as indeed it is) and looked at other lines. It turns out that there’s a further difficulty near the end of the study since 6...Kg7 7.Sxg6 Kxg6 8.Bd3+ Kh6 prevents 9.Sxb4 due to 9...Rxb4+ 10.Kg3 Rb3, and so Black again secures a winning material balance. Once again, Stockfish and the HHdbVI database would lead one to believe that the study is correct and unless it is tested against the 7-man tablebases it is hard to spot the flaw. Note that I am in no way criticising either Stockfish or Harold van der Heijden’s database. Both are terrific assets for chess generally and for studies in particular. However, it’s important to be aware of their limitations. While the database does correctly identify many studies which are unsound due to the R+B v B+S issue, there are others with the same flaw which are not so marked.

Improve this answer
0

Here's a famous study by Gijs van Breukelen which still seems to stump the engines (as of October 2023).

[Title "Gijs van Breukelen, Schakend Nederland 1990, White To Play And Win"]
[FEN "8/3P3k/n2K3p/2p3n1/1b4N1/2p1p1P1/8/3B4 w - - 1 1 "]
Improve this answer
5
  • 2
    Not sure about that - my SF16 NNUE takes about 2 minutes at ~23M nodes per second to reach depth 44 and say that White is clearly winning. The main line it gives is 1.Nf6+ Kg7 2.Nh5+ Kg6 3.Bc2+ Kxh5 4.d8Q Kg4 5.Qf6 with a score of +5.46.
    – HTTP 410
    Commented Oct 2, 2023 at 15:46
  • 2
    Note that SF's main line avoids the beautiful Q sac, wonderful mating net, and Black's desperate under-promotions (4...Nf7+ 5.Ke6 Nxd8 6.Kf5 etc) with the prosaic 4...Kg4.
    – HTTP 410
    Commented Oct 2, 2023 at 15:56
  • Nf7+ might be beautiful in the opinion of some, but objectively it deserves no better than a '?' in a sideline. Beautiful in my opinion would be if White had to sac the Q against Black's best defence and still won.
    – Rosie F
    Commented Jan 29 at 8:43
  • 1
    @RosieF 4...Nf7+ is the best defense, and this is not even an aesthetic consideration. Your engine rejects it because it is programmed to evaluate a prosaically lost position where mate arrive after 15 moves higher than an intricated but lost position where the opponent has to find 10 only moves in a row in order to win. This is not objectivity, this is a practical programming choice. Objectively, 4...Nf7!? is best.
    – Evargalo
    Commented Jan 30 at 14:48
  • 1
    @Evargalo Good point. Even engines, when they can't solve a position exactly, are in some sense declaring that the "best" move is the move that causes most trouble for a strong player. What is a 1.5-pawn advantage versus a 0.8-pawn advantage anyway? From God's point of view, every position is either totally winning for White, or totally winning for Black, or dead drawn. Except in positions where we can calculate God's algorithm, the only viable definition of "best" is in terms of the difficulties the move presents to a fallible opponent.
    – Timothy Chow
    Commented Jan 30 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.