5

Lc0 routinely makes it far in the Top Chess Engine Championship (TCEC). Latest, it defeated Komodo Dragon and nearly tied Stockfish:

Yet, we see on CCRL 40/15, Lc0 is ranked #32, 159 ELO behind Komodo!

The gap is greater for 40/2 (#24, 499) but smaller for BLITZ 2+1 (#7, 56).

I get there's hardware and maybe time control differences, but the gaps seem exaggerated.

Improve this question

2 Answers 2

Reset to default
7

They test under CPU-only conditions. CPUs are not a good platform for neural network inference (because they are designed to do well on a large variety of workloads, as opposed to more specialized hardware, which can be designed to specifically run neural networks efficiently), and therefore Leela, being an engine heavily relying on neural networks, is much weaker under CPU-only conditions than it is when having access to a good GPU. Basically, the core computational workload of neural networks comes from doing many large matrix-vector multiplications, and graphics hardware is very good at doing those efficiently.

Improve this answer
1
  • Accepted for technical explanation for broader audience.
    – OverLordGoldDragon
    Commented Aug 17, 2023 at 11:33
2

There's quite a bit to pay attention to when comparing engines: time control, hardware, opening book, etc. In the case of CCRL and Leela, this is the key line:

enter image description here

Leela is a GPU engine - it needs GPU resources to be optimal. This compares to CPU engines, which both Komodo and Stockfish are. It's why hardware was so controversial when Leela first appeared (you can't run Komodo and Stockfish on GPU, and Leela on CPU is much weaker, so how can you have a "fair" match between them?).

CCRL uses CPU to do their benchmarking, which means Leela is much weaker, and is why Leela is rated so poorly.

Improve this answer
2
  • "how can you have a "fair" match between them?" How'd TCEC figure it out? I think determining comparable compute is entirely possible, and that it's about CCRL being unable to afford hardware. Thanks -- would you know of an alternate leaderboard? (optional question)
    – OverLordGoldDragon
    Commented Aug 17, 2023 at 11:35
  • @OverLordGoldDragon There're several different ways you might define "fair". An example is "best commercially available". Another is "equal power consumption". Yet another is "equal cost". To get the exact specifics, you'd have to ask the TCEC admins. As for rating lists, CCRL is the most widely-followed one, but there's also sp-cc.de, probably some other lists as well.
    – Allure
    Commented Aug 17, 2023 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.