How does Stockfish ensure an incremental change made is beneficial?

Question

Does Stockfish's testing framework get the new engine to play the old engine many times in a match? Since Stockfish thinks on each move according to a time limit (and not a depth limit, which gives a deterministic amount of calculation), randomness in the OS' speed affects the engine's performance. This means that the match result could be more determined by randomness in the OS than by the incremental change made to Stockfish, causing the new engine to lose.

I'm working on my own engine with this testing framework, and I've found this to be a problem. Are there any other testing frameworks for engines that aren't subject to how the OS is performing? I also run matches with a depth limit for the engines, but these matches aren't ideal because in some positions an engine needs to calculate more or can get away with calculating less.

Answer 1

Disclaimer: not an expert.

As I understand the so-called "Fishtesting", you write a patch and submit it to the queue. The system then plays your patch against the previous engine at short time controls. Very many games are played (this is why computing power is needed to improve your engine). If the patch proves to be stronger at short time controls, another test is performed and longer time controls, and if the patch passes again then it's promoted to become the main version.

Example: this patch. You can see the user submitting the patch is "protonspring", and he was trying to simplify the passed pawns code. Fishtesting played the new patch against the old version 5451 times, with a result of 20.1% wins, 23.3% losses, and 56.5% draws. The new patch is inferior, so it's rejected. As you can see from the Fishtest main page, most patches fail.