Why does Fishtesting test Stockfish against itself, not other engines?

Question

As I understand the so-called Fishtesting, people write patches, then try the patches against the current version of Stockfish. If the new version performs better, it's promoted to be the main version; otherwise it's rejected.

Question: why test against the current version of Stockfish? It's worked in the past, and current Stockfish is a lot stronger than two-year old Stockfish. However, we can also see limitations to this approach. Although Stockfish is now capable of utterly destroying other traditional engines, it also recently lost the TCEC S15 superfinal against Leela. Therefore, why not test each new version against Leela?

The obvious answer is that Fishtesting's resources come from CPUs, and Leela is awful on CPU. However that shouldn't be fatal: one could just give Leela more time, e.g. with Stockfish running on 10s per game, give Leela 100s per game or whatever time odds are necessary to get the Leela ratio to the ratio in the superfinal. This will certainly slow down Fishtesting, but if it leads to an engine that can beat Leela in a match, that might still be worth it.

Answer 1

First of all, what does not become clear from other answers is that you of course can in principle use any opponent (with a consistent playing strength) to compare the playing strength of two versions of a program.

However, there are several good reasons to test the versions directly against each other, regardless of whether you are developing Stockfish or any other engine:

• The playing strength comparison of chess playing entities usually does not fulfill transitivity, i.e., if A > B and B > C, A < C is still possible. So as long as a comparison against a large pool of opponents (in order to average out the transitivity issue) is not feasible, a direct comparison should be the most reliable measurement, although you can of course have the same issue for three versions of a single program.
• By doing a direct comparison, you reduce the required number of games in testing, because you only need to run one match instead of two, and at the same time you get smaller error bars for the playing strength difference.
• Testing against an opponent of (almost) equal strength maximizes the statistical sensitivity of a single game result, so you again save hardware costs. If the difference in playing strength is very high, the information/entropy of a single game is very low.

Furthermore, in the case of fishtest/Stockfish there are several concrete reasons why choosing a direct comparison was the most principled choice:

• At the time fishtest was initially developed, Leela was not available.
• Using commercial engines such as Komodo and Houdini was/is not an option because of licensing issues.
• Using other programs can introduce security risks as well as additional software dependencies or hardware requirements you want to avoid in a distributed computing environment.
• Previous to Leela, there basically was no open source engine that was strong enough in order to get statistically significant results when matching against Stockfish.

Answer 2

The point of fishtesting is to test if a newer version of Stockfish has become stronger. The definition of stronger is it beating its previous version.

I'm not sure how testing each new version against an engine like Leela would help. Your idea might be only accepting a new version of Stockfish if it performs better against Leela than the previous version of Stockfish did. So now each version of Stockfish is guaranteed to play better against Leela, but not guaranteed to be a better chess player overall. The reason is that Stockfish could rely on certain characteristics in Leela's play to win more games.

EDIT - I thought of an example that might help here. Suppose you were rated 1500 a few years ago, and now you're 1800. If you played your past self in a match it's clear you would win. But for the sake of argument it's possible your 1500-self would perform better against Kasparov than your current 1800 self. Maybe the 1500-self was less conservative and took way more risks. This manages to beat Kasparov in a rare game, while your current 1800 self would lose every time (yet the games would be slightly closer on average).

Answer 3

I'll agree in the main with @inertialignorance but I'd like to clarify the position a bit.

When human plays human, the result of one game is relatively meaningless, given the extreme variability of human play. (I'll use Kasparov v Deep Blue as an example of that variability -- on a good day I probably could have beaten Kasparov in the game he blundered away against DB.) So let's just ignore humanity in this and concentrate on the machine.

Machines play at a far more consistent level of skill than humans. Therefore a single game means more, and a series of games means a lot. So it makes sense to test the proposed Stockfish improvement against a machine. But why Stockfish?

Simply because there's no way to quantify "does better against Leela" in any meaningful way. It could do better, and still lose. But what's the standard for better, in that case? How do you prove it played better? I can't see a viable standard.

It's far simpler to prove Stockfish prime is an improvement over Stockfish original by simply playing a set of games and counting the results.

Edward Deming maintained that it was of prime importance to chose the correct metric, because you only know you improve because of what you measure. So ask the question: What is the purpose of the process? Why are you proposing a patch for Stockfish?

Is it really the ultimate goal of the patch to beat Leela? Or is it rather to make Stockfish play better chess? I'd say it's the latter. Beating Leela will come on its own if only Stockfish continues to improve enough.

Yes, an incremental improvement path for Stockfish could come to a plateau. Even so, the only verifiable path off that plateau will lie in making changes to it that make it better. If an approach "hits a wall," then a continual search for improvements will dictate another path. If a proposed alternate path can't beat the current path, why choose it?

Answer 4

There appears to be a hardware reason for not making this change.

The main problem with using Leela as a sparring opponent is that Leela runs best on GPU. It's possible to run Leela on CPU, but Leela's performance suffers a lot. The OP suggests giving Leela time odds to compensate, but time odds does not work very well: Leela's performance weakens so much that the time odds required are oppressively long.

To get an idea of how much weaker Leela is on CPU, we can look at Leela in season 12 of TCEC, when it didn't have GPU support and ran on CPU. Here's an example game played by Leela then. If one looks at the speeds it was achieving, it's about 1-3kn/s, or 1000-3000 positions per second. Comparatively, in the latest season 15 when Leela was running on powerful GPUs, it would achieve about 50kn/s (example game). Therefore to be able to test Stockfish against Leela on equal terms, one needs to give Leela about 25x time odds. If Stockfish has one minute, Leela needs 25 minutes.

As of time of writing, Fishtesting tests at two time controls: 10s + 0.1s/move, and 60s + 0.6s/move (patches that pass the first, short time control test get promoted to the longer one and tested again. Patches that pass the second then become the "new" version). At 25x time odds, Leela needs 250s + 2.5s/move in the first time control and 1500s + 15s/move in the second time control. The slowdown is tremendous; we would effectively have Leela playing at rapid time controls and not bullet. The number of games that can be completed in unit time would also drop by about 25x. Fishtesting regularly needs tens of thousands of games to test each patch; taking 25x as long to finish each test sounds unacceptably slow.

To add fuel to the fire, as far as I know, Leela had a smaller network in season 12 - the latest nets' performance could be even slower on CPU now.

It's possible that someday, if Fishtesting gets the GPU resources to test with Leela, it might switch over; however we are not at that point yet.

Edit: Lc0-CPU is currently playing at TCEC. It gets about 5knps. According to some people in the chat though, Lc0-CPU is modified to play on CPU; unmodified, it is about 80x slower than it is on GPU. Therefore there's a real hardware cost to using Lc0-CPU as a testing opponent.

Answer 5

(I'm adding another answer because it's not actually mine)

A Stockfish developer by the name of Михаил Чалый wrote this on the Fishcooking Google Groups (basically a forum for Stockfish developers) when this same idea was suggested there:

1) We don't have hardware suitable for lc0 on most of the machines (80+% of machines are noob machines* w/o GPU);

2) testing against external engine will double error bars thus increase amount of games needing for test to be reliable by factor of 4;

3) you will need to write new logic since you a) can't use SPRT, b) you need to normalize leela/sf speed to some value (as we do with TC in normal tests) so you will need assymmetric time controls;

4) there are 0 proof that patches that work in selfplay don't work vs leela. Actually there are opposite proofs like fastgm 16 cores list where sf 11 is ahead of sf10 by 50 elo which is exactly what we measure if you match them against each other.

So tldr - it's hard or almost impossible to do and gains of this are not really existing anyway.

*As far as I'm aware, "noob machines" are machines owned by another person in the community named "noob", who has lots of CPU hardware available but not GPU hardware.

Answer 6

Inertial is correct. I should also add there was no strong open source engine before LC0. Both Komodo and Houdini had licensing restriction.