Why does stockfish move quality increase non-monotonically with increasing depth? -- selected moves at depth 3 are worse than depth 1

Question

I have been working on a project looking at how stockfish moves change with increasing computation time (here search depth), and have found something peculiar - namely that the quality of moves that stockfish selects does not increase monotonically with depth.

The question is - why are the moves that stockfish selects at depth 3 worse than the moves that stockfish selects at depth 1?

A bit on how i’ve analyzed this. I’ve taken many positions for games appearing online, and then for each position collected stockfish’s move suggestions in that position, at each sequentially increasing depth limit (so getting the move suggested at depth 1, depth 2, depth 3 etc... up to depth 18). Then I have evaluated each of those suggested moves by carrying out the move and using stockfish (at depth limit 18) to evaluate the resultant board state.

The plot below shows the average evaluations of the move stockfish selects at each increasing depth. (Note that for each position, I translated the evaluation to win probability, and then subtracted the evaluation of the move at depth 1 from the rest of the line so it would be at 0. Additionally, I have cleared the cache before each evaluation to try to remove any ordering effects).

Curiously, the evaluations are non-monotonic with respect to depth. That is, according to the evaluations of stockfish at depth 18, the moves selected by stockfish at depth 3 are worse than the moves selected at depth 1.

Would anyone understand why this happens? I would have expected, on average, move quality to increase with increasing search depth.

Answer 1

I verified your results through game matches.

d3 vs d1

Here d1 wins. I am surprise with this result.

Score of sf14.1_d3 vs sf14.1_d1: 423 - 463 - 114  [0.480] 1000
...      sf14.1_d3 playing White: 245 - 196 - 59  [0.549] 500
...      sf14.1_d3 playing Black: 178 - 267 - 55  [0.411] 500
...      White vs Black: 512 - 374 - 114  [0.569] 1000
Elo difference: -13.9 +/- 20.3, LOS: 9.0 %, DrawRatio: 11.4 %

d4 vs d1

d4 now wins.

Score of sf14.1_d4 vs sf14.1_d1: 665 - 237 - 98  [0.714] 1000
...      sf14.1_d4 playing White: 336 - 113 - 51  [0.723] 500
...      sf14.1_d4 playing Black: 329 - 124 - 47  [0.705] 500
...      White vs Black: 460 - 442 - 98  [0.509] 1000
Elo difference: 158.9 +/- 22.4, LOS: 100.0 %, DrawRatio: 9.8 %

Use sf14

d3 wins

Score of sf14_d3 vs sf14_d1: 564 - 337 - 99  [0.614] 1000
...      sf14_d3 playing White: 296 - 156 - 48  [0.640] 500
...      sf14_d3 playing Black: 268 - 181 - 51  [0.587] 500
...      White vs Black: 477 - 424 - 99  [0.526] 1000
Elo difference: 80.3 +/- 20.9, LOS: 100.0 %, DrawRatio: 9.9 %

So this has to do with the changes in Stockfish itself.

I checked the code compiling and testing every commit starting from the release of sf14, that has to do with pruning/reduction. The commit that is responsible on the d3 loss vs d1 is this: https://github.com/official-stockfish/Stockfish/commit/ad357e147a1b8481a04761d726ce1db14115a68f

d3 vs d1

The 20210828 in sf20210828_d3 is the commit date .

Score of sf20210828_d3 vs sf20210828_d1: 393 - 478 - 129  [0.458] 1000
...      sf20210828_d3 playing White: 216 - 219 - 65  [0.497] 500
...      sf20210828_d3 playing Black: 177 - 259 - 64  [0.418] 500
...      White vs Black: 475 - 396 - 129  [0.539] 1000
Elo difference: -29.6 +/- 20.2, LOS: 0.2 %, DrawRatio: 12.9 %