Does Stockfish's testing framework get the new engine to play the old engine many times in a match? Since Stockfish thinks on each move according to a time limit (and not a depth limit, which gives a deterministic amount of calculation), randomness in the OS' speed affects the engine's performance. This means that the match result could be more determined by randomness in the OS than by the incremental change made to Stockfish, causing the new engine to lose.
I'm working on my own engine with this testing framework, and I've found this to be a problem. Are there any other testing frameworks for engines that aren't subject to how the OS is performing? I also run matches with a depth limit for the engines, but these matches aren't ideal because in some positions an engine needs to calculate more or can get away with calculating less.