Alpha Zero vs Lc0 - time for self-play

Question

Alpha Zero defeated Stockfish after only 9 hours of self-play. On the other hand, Lc0, has been training for several months and while attaining impressive strength (to say the least...) it's still not crushing Stockfish. What can be the reason for such a dramatic difference in learning efficiency?

Answer 1

In short, because DeepMind has TPUs. Lots and lots of TPUs.

By far, the most time-consuming part of the training process is generating self-play games. DeepMind used 5000 TPUs to generate self-play games, which is a lot of processing power. Meanwhile, Lc0 crowdsources processing power from volunteer clients, which is much slower.

From the DeepMind paper published December 2017 (emphasis mine):

We trained a separate instance of AlphaZero for each game. Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs to generate self-play games and 64 second-generation TPUs to train the neural networks.

To achieve the level of performance that AlphaZero got against the (sub-optimally tuned) Stockfish version 8, the training run took about 9 hours and 44 million self-play games for chess:

Meanwhile, the Lc0 project got started sometime in January 2018, and by April 2018, they'd only generated 8.2 million out of the needed 44 million games. I believe they were generating about 200k games per day at the time.

They also came across a lot of bugs in the process and were figuring out a lot of things as they went, so when the first training run plateaued, the best net was not as good as AlphaZero. Since then, the Lc0 project has actually restarted the training run multiple times:

• old main: The original 192x15 "main" run, using the original lczero.exe engine. It ended in August 2018, with the last net being ID595. (These nets don't seem to be listed anymore though? The early IDs at the bottom of the net list seem to be these test nets.)
• test10: The original 256x20 test run. This was a separate pipeline using a completely rewritten lc0.exe engine that was much faster at generating self-play games. Net numbering for this run started with ID10000 on June 27, 2018, and ended with ID11262 on August 30, 2018.
• test20: A run with various changes, starting with ID20001 on August 30, 2018 and ending with ID22201 on November 16, 2018.
• test30: A run testing various new ideas, starting with ID30001 on September 9, 2018. As of January 14, 2019, the best net was ID32598.
• test40: The December 6, 2018 AlphaZero paper revealed a lot of differences between what DeepMind did and what Lc0 did, and it looks like a new test run started with ID40001 on January 8, 2019. As of January 14, 2019, the best net was ID40202.

To get a sense of how well Lc0 is doing compared to AlphaZero, it's more useful to look at the Elo with respect to total number of games generated, not time elapsed. Here's what the graph looks like for the entire Lc0 project as of January 14, 2019 (note that this is self-play Elo, which is inflated compared to actual Elo):

As you can see, they've reset the training run many times. There were also a number of bootstrap mini-resets during the original main run to fix some critical bugs, and there was also a temporary test35 to test the newly revealed configuration settings from the December 2018 AlphaGo paper. Yep, this isn't confusing at all! (/s)

There are lots of calculations and graphs in this spreadsheet here that compares lc0's performance with AlphaZero. For example, here's the graph from the first old main run (under the "Graphs" tab):

From that link, an explanation of what "Leela Max" is:

"Leela Max" is Leela's theoretical maximum performance when given hardware and/or time to the point of absolute diminished returns. It may be more comparable to AlphaZero, as AlphaZero operated on 4.8 million nodes per move, whereas a conventional setup only gets a small fraction of that.

---

These days, the self-play generation speed is much faster than it was at the beginning. As of January 14, 2019, it looks like the project is generating about 1.5 million games per day. But at this speed, it'll still take a month to achieve what DeepMind achieved in 9 hours. More like double the time, really, if the training games are being diverted towards two separate runs.

Answer 2

This is happening because AlphaZero and Leela aren't playing against the same Stockfish. If you read the paper, AlphaZero beat Stockfish 8 (it also played a series against Stockfish 9 - same logic applies however). Both these versions of Stockfish are old, and significantly inferior to the latest version of Stockfish.

AlphaZero beat Stockfish 8 by +155 -6 =839. That's a ~50 elo difference. On the other hand Stockfish 10 is better than Stockfish 8 by ~100 elo. Leela, playing against Stockfish Dev (which is even better than Stockfish 10 - the engine is continuously updated) and not Stockfish 8, could be as good as AlphaZero yet fail to crush Stockfish. It would crush Stockfish 8 alright, but not Stockfish Dev.

On a side note: it's not clear that AlphaZero would crush Stockfish 10 either, or vice versa, because elo isn't transitive: if A beats B by 50 elo and B beats C by 50 elo, it doesn't mean that A beats C by 100 elo.

Edit: To illustrate, in the TCEC cup semifinals, Leela vs. Stockfish, the version of Stockfish that played was Stockfish 270918. The last six numbers is the date - Stockfish as of the 27th of the 9th month of 2018. You can see from the link above that this version of Stockfish is roughly 100 elo stronger than Stockfish 8, so the fact that Leela performs worse against it is not a surprise.