8

I have been working on a project looking at how stockfish moves change with increasing computation time (here search depth), and have found something peculiar - namely that the quality of moves that stockfish selects does not increase monotonically with depth.

The question is - why are the moves that stockfish selects at depth 3 worse than the moves that stockfish selects at depth 1?

A bit on how i’ve analyzed this. I’ve taken many positions for games appearing online, and then for each position collected stockfish’s move suggestions in that position, at each sequentially increasing depth limit (so getting the move suggested at depth 1, depth 2, depth 3 etc... up to depth 18). Then I have evaluated each of those suggested moves by carrying out the move and using stockfish (at depth limit 18) to evaluate the resultant board state.

The plot below shows the average evaluations of the move stockfish selects at each increasing depth. (Note that for each position, I translated the evaluation to win probability, and then subtracted the evaluation of the move at depth 1 from the rest of the line so it would be at 0. Additionally, I have cleared the cache before each evaluation to try to remove any ordering effects).

enter image description here

Curiously, the evaluations are non-monotonic with respect to depth. That is, according to the evaluations of stockfish at depth 18, the moves selected by stockfish at depth 3 are worse than the moves selected at depth 1.

Would anyone understand why this happens? I would have expected, on average, move quality to increase with increasing search depth.

9
  • Depth = 18 is hardly the truth about a position. What is your nodes/sec ? Feb 8, 2022 at 5:54
  • Which stockfish version did you use?
    – ferdy
    Feb 8, 2022 at 6:29
  • 2
    why would you expect on average for a move quality to increase with increasing search depth? Feb 8, 2022 at 7:33
  • 1
    Do you impose a time limit? (A user interface to Stockfish might impose one, with a default value which is low enough that it would be typical for Stockfish's search to be incomplete.)
    – Rosie F
    Feb 8, 2022 at 11:27
  • @ferdy I used the newest version - 14.1 Feb 8, 2022 at 16:42

1 Answer 1

6

I verified your results through game matches.

d3 vs d1

Here d1 wins. I am surprise with this result.

Score of sf14.1_d3 vs sf14.1_d1: 423 - 463 - 114  [0.480] 1000
...      sf14.1_d3 playing White: 245 - 196 - 59  [0.549] 500
...      sf14.1_d3 playing Black: 178 - 267 - 55  [0.411] 500
...      White vs Black: 512 - 374 - 114  [0.569] 1000
Elo difference: -13.9 +/- 20.3, LOS: 9.0 %, DrawRatio: 11.4 %

d4 vs d1

d4 now wins.

Score of sf14.1_d4 vs sf14.1_d1: 665 - 237 - 98  [0.714] 1000
...      sf14.1_d4 playing White: 336 - 113 - 51  [0.723] 500
...      sf14.1_d4 playing Black: 329 - 124 - 47  [0.705] 500
...      White vs Black: 460 - 442 - 98  [0.509] 1000
Elo difference: 158.9 +/- 22.4, LOS: 100.0 %, DrawRatio: 9.8 %

Use sf14

d3 wins

Score of sf14_d3 vs sf14_d1: 564 - 337 - 99  [0.614] 1000
...      sf14_d3 playing White: 296 - 156 - 48  [0.640] 500
...      sf14_d3 playing Black: 268 - 181 - 51  [0.587] 500
...      White vs Black: 477 - 424 - 99  [0.526] 1000
Elo difference: 80.3 +/- 20.9, LOS: 100.0 %, DrawRatio: 9.9 %

So this has to do with the changes in Stockfish itself.

I checked the code compiling and testing every commit starting from the release of sf14, that has to do with pruning/reduction. The commit that is responsible on the d3 loss vs d1 is this: https://github.com/official-stockfish/Stockfish/commit/ad357e147a1b8481a04761d726ce1db14115a68f

d3 vs d1

The 20210828 in sf20210828_d3 is the commit date .

Score of sf20210828_d3 vs sf20210828_d1: 393 - 478 - 129  [0.458] 1000
...      sf20210828_d3 playing White: 216 - 219 - 65  [0.497] 500
...      sf20210828_d3 playing Black: 177 - 259 - 64  [0.418] 500
...      White vs Black: 475 - 396 - 129  [0.539] 1000
Elo difference: -29.6 +/- 20.2, LOS: 0.2 %, DrawRatio: 12.9 %
4
  • Thank you! Nice to know that I didn't just make an error. I still would be curious to know what changes lead to this happening. Feb 8, 2022 at 19:43
  • 1
    @EvanRussek updated the answer showing which commit defeats the d3 to d1.
    – ferdy
    Feb 9, 2022 at 0:44
  • (+1) digging to the commit level details. Feb 9, 2022 at 13:18
  • @ferdy thank you! Feb 9, 2022 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.