6

As I understand the so-called Fishtesting, people write patches, then try the patches against the current version of Stockfish. If the new version performs better, it's promoted to be the main version; otherwise it's rejected.

Question: why test against the current version of Stockfish? It's worked in the past, and current Stockfish is a lot stronger than two-year old Stockfish. However, we can also see limitations to this approach. Although Stockfish is now capable of utterly destroying other traditional engines, it also recently lost the TCEC S15 superfinal against Leela. Therefore, why not test each new version against Leela?

The obvious answer is that Fishtesting's resources come from CPUs, and Leela is awful on CPU. However that shouldn't be fatal: one could just give Leela more time, e.g. with Stockfish running on 10s per game, give Leela 100s per game or whatever time odds are necessary to get the Leela ratio to the ratio in the superfinal. This will certainly slow down Fishtesting, but if it leads to an engine that can beat Leela in a match, that might still be worth it.

5

The point of fishtesting is to test if a newer version of Stockfish has become stronger. The definition of stronger is it beating its previous version.

I'm not sure how testing each new version against an engine like Leela would help. Your idea might be only accepting a new version of Stockfish if it performs better against Leela than the previous version of Stockfish did. So now each version of Stockfish is guaranteed to play better against Leela, but not guaranteed to be a better chess player overall. The reason is that Stockfish could rely on certain characteristics in Leela's play to win more games.

EDIT - I thought of an example that might help here. Suppose you were rated 1500 a few years ago, and now you're 1800. If you played your past self in a match it's clear you would win. But for the sake of argument it's possible your 1500-self would perform better against Kasparov than your current 1800 self. Maybe the 1500-self was less conservative and took way more risks. This manages to beat Kasparov in a rare game, while your current 1800 self would lose every time (yet the games would be slightly closer on average).

  • And in the end, "Slightly closer" would be hard to quantify. Self-play is best. – Brandon_J Jun 7 at 13:31
  • 1
    Seems like semantics, but what makes you say "stronger" is defined as beating the previous version? One could equally argue that "stronger" is the guy that beat Kasparov, wins the tournament, and gets all the glory, not the one that wins the head-to-head. – Allure Jun 7 at 15:13
  • 1
    @Allure I think head-to-head is just the most objective measure of "strength". If A beats B, A played better than B, ergo A is stronger than B. But if A does slightly less worse against C than B did against C, does that necessarily mean A > B? What if B performed better against D, E, F, G... than A did? – Inertial Ignorance Jun 7 at 18:49
  • 1
    But you have a point. The Candidates system for the World Championship is a big tournament, but there used to be individual matches between players. For a big tournament A could lose to B but still win the tournament and be declared "strongest". However, this is because A beat a variety of different players, not just one player C. So maybe engine testing could be used where the newer version plays a variety of different engines, but it's faster and simpler to just test against the older version. – Inertial Ignorance Jun 7 at 18:51
3

I'll agree in the main with @inertialignorance but I'd like to clarify the position a bit.

When human plays human, the result of one game is relatively meaningless, given the extreme variability of human play. (I'll use Kasparov v Deep Blue as an example of that variability -- on a good day I probably could have beaten Kasparov in the game he blundered away against DB.) So let's just ignore humanity in this and concentrate on the machine.

Machines play at a far more consistent level of skill than humans. Therefore a single game means more, and a series of games means a lot. So it makes sense to test the proposed Stockfish improvement against a machine. But why Stockfish?

Simply because there's no way to quantify "does better against Leela" in any meaningful way. It could do better, and still lose. But what's the standard for better, in that case? How do you prove it played better? I can't see a viable standard.

It's far simpler to prove Stockfish prime is an improvement over Stockfish original by simply playing a set of games and counting the results.

Edward Deming maintained that it was of prime importance to chose the correct metric, because you only know you improve because of what you measure. So ask the question: What is the purpose of the process? Why are you proposing a patch for Stockfish?

Is it really the ultimate goal of the patch to beat Leela? Or is it rather to make Stockfish play better chess? I'd say it's the latter. Beating Leela will come on its own if only Stockfish continues to improve enough.

Yes, an incremental improvement path for Stockfish could come to a plateau. Even so, the only verifiable path off that plateau will lie in making changes to it that make it better. If an approach "hits a wall," then a continual search for improvements will dictate another path. If a proposed alternate path can't beat the current path, why choose it?

  • How do you prove it played better? it should be pretty straightforward, no? Just play the previous version 20k times against Leela, and the new version 20k times as well, then compare the results. – Allure Jun 11 at 22:19
  • @Allure But hypothetically, what if Stockfish Prime beat old Stockfish, but old Stockfish did better against Leela (and as Arlen mentioned, how do you even compare which Stockfish "did better" against Leela). Would you not keep Stockfish prime then? This verification process seems flawed. – Inertial Ignorance Jun 12 at 0:30
  • @InertialIgnorance Equally, what if Stockfish Prime did better against Leela, but failed to beat old Stockfish, would you not keep Stockfish Prime then? It comes down to how one defines "stronger", and it's not clear to me that "stronger" is winning the head-to-head. – Allure Jun 12 at 0:36
  • @Allure I'll admit there's still a reason to be conflicted there, but in that case it feels more clear to prefer Stockfish Prime. When you want to compare which two things are better (be them politicians, athletes, etc), you most often get them to compete against each other in some fashion. Not to see who did better against an arbitrarily selected peer. – Inertial Ignorance Jun 12 at 2:55
  • @InertialIgnorance still, if Karjakin had won the 2016 World Championships, it's hard for me to imagine that most people will consider him the strongest player in the world. Yes, he'd have beaten Carlsen in the head-to-head match, but Carlsen's tournament victories would indicate the reverse. Similarly when Kasparov wasn't world champion, most people still considered him the strongest. – Allure Jun 12 at 4:01
2

There appears to be a hardware reason for not making this change.

The main problem with using Leela as a sparring opponent is that Leela runs best on GPU. It's possible to run Leela on CPU, but Leela's performance suffers a lot. The OP suggests giving Leela time odds to compensate, but time odds does not work very well: Leela's performance weakens so much that the time odds required are oppressively long.

To get an idea of how much weaker Leela is on CPU, we can look at Leela in season 12 of TCEC, when it didn't have GPU support and ran on CPU. Here's an example game played by Leela then. If one looks at the speeds it was achieving, it's about 1-3kn/s, or 1000-3000 positions per second. Comparatively, in the latest season 15 when Leela was running on powerful GPUs, it would achieve about 50kn/s (example game). Therefore to be able to test Stockfish against Leela on equal terms, one needs to give Leela about 25x time odds. If Stockfish has one minute, Leela needs 25 minutes.

As of time of writing, Fishtesting tests at two time controls: 10s + 0.1s/move, and 60s + 0.6s/move (patches that pass the first, short time control test get promoted to the longer one and tested again. Patches that pass the second then become the "new" version). At 25x time odds, Leela needs 250s + 2.5s/move in the first time control and 1500s + 15s/move in the second time control. The slowdown is tremendous; we would effectively have Leela playing at rapid time controls and not bullet. The number of games that can be completed in unit time would also drop by about 25x. Fishtesting regularly needs tens of thousands of games to test each patch; taking 25x as long to finish each test sounds unacceptably slow.

To add fuel to the fire, as far as I know, Leela had a smaller network in season 12 - the latest nets' performance could be even slower on CPU now.

It's possible that someday, if Fishtesting gets the GPU resources to test with Leela, it might switch over; however we are not at that point yet.

  • I doubt that this will happen. Almost all engine authors are mainly testing their engine against itself and not against other (possibly stronger) engines. It might be valuable to in addition do cross-checks by testing against other engines, but this should not be the main part of testing, because it is less efficient. – Fabian Fichter 1 hour ago
2

First of all, what does not become clear from other answers is that you of course can in principle use any opponent (with a consistent playing strength) to compare the playing strength of two versions of a program.

However, there are several good reasons to test the versions directly against each other, regardless of whether you are developing Stockfish or any other engine:

  • The playing strength comparison of chess playing entities usually does not fulfill transitivity, i.e., if A > B and B > C, A < C is still possible. So as long as a comparison against a large pool of opponents (in order to average out the transitivity issue) is not feasible, a direct comparison should be the most reliable measurement, although you can of course have the same issue for three versions of a single program.
  • By doing a direct comparison, you reduce the required number of games in testing, because you only need to run one match instead of two, and at the same time you get smaller error bars for the playing strength difference.
  • Testing against an opponent of (almost) equal strength maximizes the statistical sensitivity of a single game result, so you again save hardware costs. If the difference in playing strength is very high, the information/entropy of a single game is very low.

Furthermore, in the case of fishtest/Stockfish there are several concrete reasons why choosing a direct comparison was the most principled choice:

  • At the time fishtest was initially developed, Leela was not available.
  • Using commercial engines such as Komodo and Houdini was/is not an option because of licensing issues.
  • Using other programs can introduce security risks as well as additional software dependencies or hardware requirements you want to avoid in a distributed computing environment.
  • Previous to Leela, there basically was no open source engine that was strong enough in order to get statistically significant results when matching against Stockfish.
0

Inertial is correct. I should also add there was no strong open source engine before LC0. Both Komodo and Houdini had licensing restriction.

  • "no strong open source engine" I suppose you mean besides stockfish? – Brandon_J Jun 10 at 0:58
  • @Brandon_J yes that was what I meant – SmallChess Jun 11 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.