7

I have been working on a project looking at how stockfish moves change with increasing computation time (here search depth), and have found something peculiar - namely that the quality of moves that stockfish selects does not increase monotonically with depth.

The question is - why are the moves that stockfish selects at depth 3 worse than the moves that stockfish selects at depth 1?

A bit on how i’ve analyzed this. I’ve taken many positions for games appearing online, and then for each position collected stockfish’s move suggestions in that position, at each sequentially increasing depth limit (so getting the move suggested at depth 1, depth 2, depth 3 etc... up to depth 18). Then I have evaluated each of those suggested moves by carrying out the move and using stockfish (at depth limit 18) to evaluate the resultant board state.

The plot below shows the average evaluations of the move stockfish selects at each increasing depth. (Note that for each position, I translated the evaluation to win probability, and then subtracted the evaluation of the move at depth 1 from the rest of the line so it would be at 0. Additionally, I have cleared the cache before each evaluation to try to remove any ordering effects).

enter image description here

Curiously, the evaluations are non-monotonic with respect to depth. That is, according to the evaluations of stockfish at depth 18, the moves selected by stockfish at depth 3 are worse than the moves selected at depth 1.

Would anyone understand why this happens? I would have expected, on average, move quality to increase with increasing search depth.

9
  • Depth = 18 is hardly the truth about a position. What is your nodes/sec ? Feb 8 at 5:54
  • Which stockfish version did you use?
    – ferdy
    Feb 8 at 6:29
  • 2
    why would you expect on average for a move quality to increase with increasing search depth? Feb 8 at 7:33
  • 1
    Do you impose a time limit? (A user interface to Stockfish might impose one, with a default value which is low enough that it would be typical for Stockfish's search to be incomplete.)
    – Rosie F
    Feb 8 at 11:27
  • @ferdy I used the newest version - 14.1 Feb 8 at 16:42

1 Answer 1

5

I verified your results through game matches.

d3 vs d1

Here d1 wins. I am surprise with this result.

Score of sf14.1_d3 vs sf14.1_d1: 423 - 463 - 114  [0.480] 1000
...      sf14.1_d3 playing White: 245 - 196 - 59  [0.549] 500
...      sf14.1_d3 playing Black: 178 - 267 - 55  [0.411] 500
...      White vs Black: 512 - 374 - 114  [0.569] 1000
Elo difference: -13.9 +/- 20.3, LOS: 9.0 %, DrawRatio: 11.4 %

d4 vs d1

d4 now wins.

Score of sf14.1_d4 vs sf14.1_d1: 665 - 237 - 98  [0.714] 1000
...      sf14.1_d4 playing White: 336 - 113 - 51  [0.723] 500
...      sf14.1_d4 playing Black: 329 - 124 - 47  [0.705] 500
...      White vs Black: 460 - 442 - 98  [0.509] 1000
Elo difference: 158.9 +/- 22.4, LOS: 100.0 %, DrawRatio: 9.8 %

Use sf14

d3 wins

Score of sf14_d3 vs sf14_d1: 564 - 337 - 99  [0.614] 1000
...      sf14_d3 playing White: 296 - 156 - 48  [0.640] 500
...      sf14_d3 playing Black: 268 - 181 - 51  [0.587] 500
...      White vs Black: 477 - 424 - 99  [0.526] 1000
Elo difference: 80.3 +/- 20.9, LOS: 100.0 %, DrawRatio: 9.9 %

So this has to do with the changes in Stockfish itself.

I checked the code compiling and testing every commit starting from the release of sf14, that has to do with pruning/reduction. The commit that is responsible on the d3 loss vs d1 is this: https://github.com/official-stockfish/Stockfish/commit/ad357e147a1b8481a04761d726ce1db14115a68f

d3 vs d1

The 20210828 in sf20210828_d3 is the commit date .

Score of sf20210828_d3 vs sf20210828_d1: 393 - 478 - 129  [0.458] 1000
...      sf20210828_d3 playing White: 216 - 219 - 65  [0.497] 500
...      sf20210828_d3 playing Black: 177 - 259 - 64  [0.418] 500
...      White vs Black: 475 - 396 - 129  [0.539] 1000
Elo difference: -29.6 +/- 20.2, LOS: 0.2 %, DrawRatio: 12.9 %
4
  • Thank you! Nice to know that I didn't just make an error. I still would be curious to know what changes lead to this happening. Feb 8 at 19:43
  • 1
    @EvanRussek updated the answer showing which commit defeats the d3 to d1.
    – ferdy
    Feb 9 at 0:44
  • (+1) digging to the commit level details. Feb 9 at 13:18
  • @ferdy thank you! Feb 9 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.