10

Chess engines are so strong that it is becoming increasingly hard to find positions that they are demonstrably evaluating incorrectly. I'm interested in collecting examples—if there are any—of positions where we know the correct evaluation, but that are very difficult for engines.

Below is a beautiful study that I encountered recently in Chapter 5 of the book Barrycades and Septoku: Papers in Honor of Martin Gardner and Tom Rodgers, by Thane Plambeck and Tomas Rokicki (eds.), MAA Press, 2020.

[Title "Carlos Pereira dos Santos, White To Play And Win"]
[FEN "bR2r3/2nK4/kp1pp1pp/p1R2p2/1P1P1P1P/p7/P2P3P/8 w - - 0 1"]

The engines I have access to are old and out of date, but after 1.b5+ Ka7 they want to play 2.Rxe8? and miss the winning continuation 2.Rxa8! Nxa8 3.Rc8! Rxc8 4.Kxc8 with a win for White. I would be curious to know if stronger engines are able to find this win. [EDIT: Thanks to Oscar Smith for pointing out that I can download Stockfish 11 for free. It had some difficulty at first, but after searching about 2.5 billion nodes it finally found 2.Rxa8!]

Years ago, there was a ChessBase article on this topic. They give some examples of positions, such as fortresses, where the engines' numerical evaluation is way off. However, sometimes the engine plays the position correctly anyway, finding the drawing moves even though it thinks it's losing, and such examples are not so interesting to me. What is more interesting to me is a position where the engine loses a position that a perfect player would draw, or draws (or even loses) a position that a perfect player would win.

Probably the best example from that article is a study by Behting, shown below.

[Title "K. K. Behting, Baltische Schachblätter 1908, White To Play And Draw"]
[FEN "8/8/7p/3KNN1k/2p4p/8/3P2p1/8 w - - 0 1"]

As of 2012, careful analysis demonstrated that the study is sound but that engines had serious trouble with it. Is that still true today? [EDIT: It seems so; after searching 25 billion nodes, Stockfish 11 was still unsuccessfully trying the 1.Ng7+ line that is analyzed in detail here.]

There is one other type of example that I know about, namely tablebase positions where White has an extremely subtle win. For example, below is a position that I found on Wikipedia.

[Title "Lomonosov Tablebase, White To Play And Win"]
[FEN "8/r6n/2Q5/8/5k2/3K4/7N/3b3q w - - 0 1"]

Clearly, if an engine without tablebases plays White and an engine with tablebases plays Black, the engine playing White is not going to win. Sooner or later it is bound to blunder away the win. I would be curious if there are any humanly comprehensible examples of this type.

In the process of typing up this question, I found another question on this site that asks a very similar question. Unfortunately, some of the answers there seem to be (no longer?) valid. Also, let me emphasize again that for my purposes, a fortress position that the engine thinks is losing but that it plays correctly anyway does not count.

In short, my question is this:

Are there other good examples of positions that stump engines?

Improve this question
2
  • 5
    One thing to note is that there is no reason to not have access to top tier new engines. SF and Lc0 are the two best engines, and both are open source and relatively easy to download.
    – Oscar Smith
    May 26, 2020 at 15:57
  • @OscarSmith : Thanks for the suggestion! I have downloaded Stockfish 11 and have added a couple of comments accordingly.
    – Timothy Chow
    May 27, 2020 at 19:56

2 Answers 2

Reset to default
5

There are plenty of positions that stump engines, and it's not too difficult to find some from engine tournaments. Example, this position stumped Stockfish:

[FEN "1k1r4/pp4pp/4pb2/2Q5/2P5/8/P5PP/6K1 w - - 1 28"]

Stockfish actually reached +2.76 eval later, even though White has no way to break through. In fact Black can afford to give up the e6-pawn and the position is still drawn. Lc0 evaluated the position accurately as a draw, which doesn't mean Lc0 is some kind of goddess at evaluating fortresses, since this position stumped Lc0:

[FEN "5k2/5p2/p3pPp1/3pN2p/b6P/2P5/P5P1/7K w - - 0 35"]

Black plays ...Bb5 and defends along the f1-a6 diagonal, and White will never get anywhere, especially since if the Knight moves too much Black advances the e6-pawn with counterplay. This time Stockfish accurately assessed the position as drawn, while Lc0 was at +1.6 - or about 60% chance to win - or so for quite a while.

If you're wondering about positions with study-like solutions that humans think is pretty - that's a different question.

Improve this answer
2
  • 1
    Thanks for the examples. But if I understand correctly, the engines play these positions correctly even though they misevaluate them, right? That is, the position is actually a draw, and the engine does not throw away the draw by poor play?
    – Timothy Chow
    May 27, 2020 at 4:40
  • 2
    @TimothyChow yes, the engine does not throw away the draw by poor play.
    – Allure
    May 27, 2020 at 4:41
0

Here's a famous study by Gijs van Breukelen which still seems to stump the engines (as of October 2023).

[Title "Gijs van Breukelen, Schakend Nederland 1990, White To Play And Win"]
[FEN "8/3P3k/n2K3p/2p3n1/1b4N1/2p1p1P1/8/3B4 w - - 1 1 "]
Improve this answer
2
  • 1
    Not sure about that - my SF16 NNUE takes about 2 minutes at ~23M nodes per second to reach depth 44 and say that White is clearly winning. The main line it gives is 1.Nf6+ Kg7 2.Nh5+ Kg6 3.Bc2+ Kxh5 4.d8Q Kg4 5.Qf6 with a score of +5.46.
    – HTTP 410
    Oct 2 at 15:46
  • 1
    Note that SF's main line avoids the beautiful Q sac, wonderful mating net, and Black's desperate under-promotions (4...Nf7+ 5.Ke6 Nxd8 6.Kf5 etc) with the prosaic 4...Kg4.
    – HTTP 410
    Oct 2 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.