As I understand the so-called Fishtesting, people write patches, then try the patches against the current version of Stockfish. If the new version performs better, it's promoted to be the main version; otherwise it's rejected.

Question: why test against the current version of Stockfish? It's worked in the past, and current Stockfish is a lot stronger than two-year old Stockfish. However, we can also see limitations to this approach. Although Stockfish is now capable of utterly destroying other traditional engines, it also recently lost the TCEC S15 superfinal against Leela. Therefore, why not test each new version against Leela?

The obvious answer is that Fishtesting's resources come from CPUs, and Leela is awful on CPU. However that shouldn't be fatal: one could just give Leela more time, e.g. with Stockfish running on 10s per game, give Leela 100s per game or whatever time odds are necessary to get the Leela ratio to the ratio in the superfinal. This will certainly slow down Fishtesting, but if it leads to an engine that can beat Leela in a match, that might still be worth it.

  • Most likely because Stockfish is easily available and modifyable to Stockfish developers. Dec 18, 2022 at 18:37

6 Answers 6


First of all, what does not become clear from other answers is that you of course can in principle use any opponent (with a consistent playing strength) to compare the playing strength of two versions of a program.

However, there are several good reasons to test the versions directly against each other, regardless of whether you are developing Stockfish or any other engine:

  • The playing strength comparison of chess playing entities usually does not fulfill transitivity, i.e., if A > B and B > C, A < C is still possible. So as long as a comparison against a large pool of opponents (in order to average out the transitivity issue) is not feasible, a direct comparison should be the most reliable measurement, although you can of course have the same issue for three versions of a single program.
  • By doing a direct comparison, you reduce the required number of games in testing, because you only need to run one match instead of two, and at the same time you get smaller error bars for the playing strength difference.
  • Testing against an opponent of (almost) equal strength maximizes the statistical sensitivity of a single game result, so you again save hardware costs. If the difference in playing strength is very high, the information/entropy of a single game is very low.

Furthermore, in the case of fishtest/Stockfish there are several concrete reasons why choosing a direct comparison was the most principled choice:

  • At the time fishtest was initially developed, Leela was not available.
  • Using commercial engines such as Komodo and Houdini was/is not an option because of licensing issues.
  • Using other programs can introduce security risks as well as additional software dependencies or hardware requirements you want to avoid in a distributed computing environment.
  • Previous to Leela, there basically was no open source engine that was strong enough in order to get statistically significant results when matching against Stockfish.
  • Are you sure testing against a 3rd engine increases the number of games required? It seems like it only does so the first time. For example let's say the third engine is E, and the versions of Stockfish being tried are A, B and C. We play A and B 20k times against E and get a score. Whichever does better is kept and the weaker one rejected. Say A is kept. When it comes to testing version C, we already have the A-E results, so we only need to get C-E results, and the same applies to all future versions.
    – Allure
    Jun 17, 2019 at 3:56
  • 4
    Even if you always test against the same version of your reference engine (which makes it very prone to overfitting), the statistical uncertainty on the resulting Elo difference will still be bigger, because by combining the results from two runs (e.g., EloC - EloB) you also need to combine the statistical uncertainties, which usually increases the uncertainty by a factor of sqrt(2). In order to compensate for that, you would need to run more games. Jun 17, 2019 at 9:08

The point of fishtesting is to test if a newer version of Stockfish has become stronger. The definition of stronger is it beating its previous version.

I'm not sure how testing each new version against an engine like Leela would help. Your idea might be only accepting a new version of Stockfish if it performs better against Leela than the previous version of Stockfish did. So now each version of Stockfish is guaranteed to play better against Leela, but not guaranteed to be a better chess player overall. The reason is that Stockfish could rely on certain characteristics in Leela's play to win more games.

EDIT - I thought of an example that might help here. Suppose you were rated 1500 a few years ago, and now you're 1800. If you played your past self in a match it's clear you would win. But for the sake of argument it's possible your 1500-self would perform better against Kasparov than your current 1800 self. Maybe the 1500-self was less conservative and took way more risks. This manages to beat Kasparov in a rare game, while your current 1800 self would lose every time (yet the games would be slightly closer on average).

  • And in the end, "Slightly closer" would be hard to quantify. Self-play is best.
    – Brandon_J
    Jun 7, 2019 at 13:31
  • 1
    Seems like semantics, but what makes you say "stronger" is defined as beating the previous version? One could equally argue that "stronger" is the guy that beat Kasparov, wins the tournament, and gets all the glory, not the one that wins the head-to-head.
    – Allure
    Jun 7, 2019 at 15:13
  • 1
    @Allure I think head-to-head is just the most objective measure of "strength". If A beats B, A played better than B, ergo A is stronger than B. But if A does slightly less worse against C than B did against C, does that necessarily mean A > B? What if B performed better against D, E, F, G... than A did? Jun 7, 2019 at 18:49
  • 1
    But you have a point. The Candidates system for the World Championship is a big tournament, but there used to be individual matches between players. For a big tournament A could lose to B but still win the tournament and be declared "strongest". However, this is because A beat a variety of different players, not just one player C. So maybe engine testing could be used where the newer version plays a variety of different engines, but it's faster and simpler to just test against the older version. Jun 7, 2019 at 18:51

I'll agree in the main with @inertialignorance but I'd like to clarify the position a bit.

When human plays human, the result of one game is relatively meaningless, given the extreme variability of human play. (I'll use Kasparov v Deep Blue as an example of that variability -- on a good day I probably could have beaten Kasparov in the game he blundered away against DB.) So let's just ignore humanity in this and concentrate on the machine.

Machines play at a far more consistent level of skill than humans. Therefore a single game means more, and a series of games means a lot. So it makes sense to test the proposed Stockfish improvement against a machine. But why Stockfish?

Simply because there's no way to quantify "does better against Leela" in any meaningful way. It could do better, and still lose. But what's the standard for better, in that case? How do you prove it played better? I can't see a viable standard.

It's far simpler to prove Stockfish prime is an improvement over Stockfish original by simply playing a set of games and counting the results.

Edward Deming maintained that it was of prime importance to chose the correct metric, because you only know you improve because of what you measure. So ask the question: What is the purpose of the process? Why are you proposing a patch for Stockfish?

Is it really the ultimate goal of the patch to beat Leela? Or is it rather to make Stockfish play better chess? I'd say it's the latter. Beating Leela will come on its own if only Stockfish continues to improve enough.

Yes, an incremental improvement path for Stockfish could come to a plateau. Even so, the only verifiable path off that plateau will lie in making changes to it that make it better. If an approach "hits a wall," then a continual search for improvements will dictate another path. If a proposed alternate path can't beat the current path, why choose it?

  • How do you prove it played better? it should be pretty straightforward, no? Just play the previous version 20k times against Leela, and the new version 20k times as well, then compare the results.
    – Allure
    Jun 11, 2019 at 22:19
  • @Allure But hypothetically, what if Stockfish Prime beat old Stockfish, but old Stockfish did better against Leela (and as Arlen mentioned, how do you even compare which Stockfish "did better" against Leela). Would you not keep Stockfish prime then? This verification process seems flawed. Jun 12, 2019 at 0:30
  • @InertialIgnorance Equally, what if Stockfish Prime did better against Leela, but failed to beat old Stockfish, would you not keep Stockfish Prime then? It comes down to how one defines "stronger", and it's not clear to me that "stronger" is winning the head-to-head.
    – Allure
    Jun 12, 2019 at 0:36
  • @Allure I'll admit there's still a reason to be conflicted there, but in that case it feels more clear to prefer Stockfish Prime. When you want to compare which two things are better (be them politicians, athletes, etc), you most often get them to compete against each other in some fashion. Not to see who did better against an arbitrarily selected peer. Jun 12, 2019 at 2:55
  • @InertialIgnorance still, if Karjakin had won the 2016 World Championships, it's hard for me to imagine that most people will consider him the strongest player in the world. Yes, he'd have beaten Carlsen in the head-to-head match, but Carlsen's tournament victories would indicate the reverse. Similarly when Kasparov wasn't world champion, most people still considered him the strongest.
    – Allure
    Jun 12, 2019 at 4:01

There appears to be a hardware reason for not making this change.

The main problem with using Leela as a sparring opponent is that Leela runs best on GPU. It's possible to run Leela on CPU, but Leela's performance suffers a lot. The OP suggests giving Leela time odds to compensate, but time odds does not work very well: Leela's performance weakens so much that the time odds required are oppressively long.

To get an idea of how much weaker Leela is on CPU, we can look at Leela in season 12 of TCEC, when it didn't have GPU support and ran on CPU. Here's an example game played by Leela then. If one looks at the speeds it was achieving, it's about 1-3kn/s, or 1000-3000 positions per second. Comparatively, in the latest season 15 when Leela was running on powerful GPUs, it would achieve about 50kn/s (example game). Therefore to be able to test Stockfish against Leela on equal terms, one needs to give Leela about 25x time odds. If Stockfish has one minute, Leela needs 25 minutes.

As of time of writing, Fishtesting tests at two time controls: 10s + 0.1s/move, and 60s + 0.6s/move (patches that pass the first, short time control test get promoted to the longer one and tested again. Patches that pass the second then become the "new" version). At 25x time odds, Leela needs 250s + 2.5s/move in the first time control and 1500s + 15s/move in the second time control. The slowdown is tremendous; we would effectively have Leela playing at rapid time controls and not bullet. The number of games that can be completed in unit time would also drop by about 25x. Fishtesting regularly needs tens of thousands of games to test each patch; taking 25x as long to finish each test sounds unacceptably slow.

To add fuel to the fire, as far as I know, Leela had a smaller network in season 12 - the latest nets' performance could be even slower on CPU now.

It's possible that someday, if Fishtesting gets the GPU resources to test with Leela, it might switch over; however we are not at that point yet.

Edit: Lc0-CPU is currently playing at TCEC. It gets about 5knps. According to some people in the chat though, Lc0-CPU is modified to play on CPU; unmodified, it is about 80x slower than it is on GPU. Therefore there's a real hardware cost to using Lc0-CPU as a testing opponent.

  • I doubt that this will happen. Almost all engine authors are mainly testing their engine against itself and not against other (possibly stronger) engines. It might be valuable to in addition do cross-checks by testing against other engines, but this should not be the main part of testing, because it is less efficient. Jun 16, 2019 at 11:01
  • @FabianFichter why is it less efficient to test against other engines?
    – Allure
    Jun 16, 2019 at 12:12
  • As I described in my answer (chess.stackexchange.com/a/24714/15415) it is less efficient because you need more games (i.e., more hardware or time) to reach the same statistical precision for your Elo measurement. Jun 16, 2019 at 13:45

(I'm adding another answer because it's not actually mine)

A Stockfish developer by the name of Михаил Чалый wrote this on the Fishcooking Google Groups (basically a forum for Stockfish developers) when this same idea was suggested there:

1) We don't have hardware suitable for lc0 on most of the machines (80+% of machines are noob machines* w/o GPU);

2) testing against external engine will double error bars thus increase amount of games needing for test to be reliable by factor of 4;

3) you will need to write new logic since you a) can't use SPRT, b) you need to normalize leela/sf speed to some value (as we do with TC in normal tests) so you will need assymmetric time controls;

4) there are 0 proof that patches that work in selfplay don't work vs leela. Actually there are opposite proofs like fastgm 16 cores list where sf 11 is ahead of sf10 by 50 elo which is exactly what we measure if you match them against each other.

So tldr - it's hard or almost impossible to do and gains of this are not really existing anyway.

*As far as I'm aware, "noob machines" are machines owned by another person in the community named "noob", who has lots of CPU hardware available but not GPU hardware.


Inertial is correct. I should also add there was no strong open source engine before LC0. Both Komodo and Houdini had licensing restriction.

  • "no strong open source engine" I suppose you mean besides stockfish?
    – Brandon_J
    Jun 10, 2019 at 0:58
  • @Brandon_J yes that was what I meant
    – SmallChess
    Jun 11, 2019 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.