8

Chess engines are so strong that it is becoming increasingly hard to find positions that they are demonstrably evaluating incorrectly. I'm interested in collecting examples—if there are any—of positions where we know the correct evaluation, but that are very difficult for engines.

Below is a beautiful study that I encountered recently in Chapter 5 of the book Barrycades and Septoku: Papers in Honor of Martin Gardner and Tom Rodgers, by Thane Plambeck and Tomas Rokicki (eds.), MAA Press, 2020.

[Title "Carlos Pereira dos Santos, White To Play And Win"]
[FEN "bR2r3/2nK4/kp1pp1pp/p1R2p2/1P1P1P1P/p7/P2P3P/8 w - - 0 1"]

The engines I have access to are old and out of date, but after 1.b5+ Ka7 they want to play 2.Rxe8? and miss the winning continuation 2.Rxa8! Nxa8 3.Rc8! Rxc8 4.Kxc8 with a win for White. I would be curious to know if stronger engines are able to find this win. [EDIT: Thanks to Oscar Smith for pointing out that I can download Stockfish 11 for free. It had some difficulty at first, but after searching about 2.5 billion nodes it finally found 2.Rxa8!]

Years ago, there was a ChessBase article on this topic. They give some examples of positions, such as fortresses, where the engines' numerical evaluation is way off. However, sometimes the engine plays the position correctly anyway, finding the drawing moves even though it thinks it's losing, and such examples are not so interesting to me. What is more interesting to me is a position where the engine loses a position that a perfect player would draw, or draws (or even loses) a position that a perfect player would win.

Probably the best example from that article is a study by Behting, shown below.

[Title "K. K. Behting, Baltische Schachblätter 1908, White To Play And Draw"]
[FEN "8/8/7p/3KNN1k/2p4p/8/3P2p1/8 w - - 0 1"]

As of 2012, careful analysis demonstrated that the study is sound but that engines had serious trouble with it. Is that still true today? [EDIT: It seems so; after searching 25 billion nodes, Stockfish 11 was still unsuccessfully trying the 1.Ng7+ line that is analyzed in detail here.]

There is one other type of example that I know about, namely tablebase positions where White has an extremely subtle win. For example, below is a position that I found on Wikipedia.

[Title "Lomonosov Tablebase, White To Play And Win"]
[FEN "8/r6n/2Q5/8/5k2/3K4/7N/3b3q w - - 0 1"]

Clearly, if an engine without tablebases plays White and an engine with tablebases plays Black, the engine playing White is not going to win. Sooner or later it is bound to blunder away the win. I would be curious if there are any humanly comprehensible examples of this type.

In the process of typing up this question, I found another question on this site that asks a very similar question. Unfortunately, some of the answers there seem to be (no longer?) valid. Also, let me emphasize again that for my purposes, a fortress position that the engine thinks is losing but that it plays correctly anyway does not count.

In short, my question is this:

Are there other good examples of positions that stump engines?

  • 4
    One thing to note is that there is no reason to not have access to top tier new engines. SF and Lc0 are the two best engines, and both are open source and relatively easy to download. – Oscar Smith May 26 at 15:57
  • @OscarSmith : Thanks for the suggestion! I have downloaded Stockfish 11 and have added a couple of comments accordingly. – Timothy Chow May 27 at 19:56
4

There are plenty of positions that stump engines, and it's not too difficult to find some from engine tournaments. Example, this position stumped Stockfish:

[FEN "1k1r4/pp4pp/4pb2/2Q5/2P5/8/P5PP/6K1 w - - 1 28"]

Stockfish actually reached +2.76 eval later, even though White has no way to break through. In fact Black can afford to give up the e6-pawn and the position is still drawn. Lc0 evaluated the position accurately as a draw, which doesn't mean Lc0 is some kind of goddess at evaluating fortresses, since this position stumped Lc0:

[FEN "5k2/5p2/p3pPp1/3pN2p/b6P/2P5/P5P1/7K w - - 0 35"]

Black plays ...Bb5 and defends along the f1-a6 diagonal, and White will never get anywhere, especially since if the Knight moves too much Black advances the e6-pawn with counterplay. This time Stockfish accurately assessed the position as drawn, while Lc0 was at +1.6 - or about 60% chance to win - or so for quite a while.

If you're wondering about positions with study-like solutions that humans think is pretty - that's a different question.

| improve this answer | |
  • Thanks for the examples. But if I understand correctly, the engines play these positions correctly even though they misevaluate them, right? That is, the position is actually a draw, and the engine does not throw away the draw by poor play? – Timothy Chow May 27 at 4:40
  • @TimothyChow yes, the engine does not throw away the draw by poor play. – Allure May 27 at 4:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.