I have been working on a project looking at how stockfish moves change with increasing computation time (here search depth), and have found something peculiar - namely that the quality of moves that stockfish selects does not increase monotonically with depth.
The question is - why are the moves that stockfish selects at depth 3 worse than the moves that stockfish selects at depth 1?
A bit on how i’ve analyzed this. I’ve taken many positions for games appearing online, and then for each position collected stockfish’s move suggestions in that position, at each sequentially increasing depth limit (so getting the move suggested at depth 1, depth 2, depth 3 etc... up to depth 18). Then I have evaluated each of those suggested moves by carrying out the move and using stockfish (at depth limit 18) to evaluate the resultant board state.
The plot below shows the average evaluations of the move stockfish selects at each increasing depth. (Note that for each position, I translated the evaluation to win probability, and then subtracted the evaluation of the move at depth 1 from the rest of the line so it would be at 0. Additionally, I have cleared the cache before each evaluation to try to remove any ordering effects).
Curiously, the evaluations are non-monotonic with respect to depth. That is, according to the evaluations of stockfish at depth 18, the moves selected by stockfish at depth 3 are worse than the moves selected at depth 1.
Would anyone understand why this happens? I would have expected, on average, move quality to increase with increasing search depth.