I understand that AlphaZero has to use a different kind of hardware than regular Stockfish. I would expect that the hardware has a large effect on engine strength. That's why I wonder whether there have been any attempts made to provide comparable hardware to both. Also what would "comparable" mean here?

Specifically I read that people complain about:

  • Stockfish being given only 1 GB of cache, and
  • the time limit of 1 min/move (How would this disadvantage Stockfish?)
  • Hash size, not cache, apparently. – Federico Poloni Dec 9 '17 at 17:15
  • 2
    I strongly suggest asking this on a technical forum (like the AI Stack Exchange, perhaps titled "Fairness of evaluation in AlphaZero vs. Stockfish matches"), because the answers here are not good. – Veedrac Dec 9 '17 at 23:14
up vote 20 down vote accepted

That's why I wonder whether there have been any attempts made to provide comparable hardware to both.

This is Google you're talking about! So the answer is obviously "No".

From the original paper hardware used for initialising and training -

Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation TPUs to train the neural networks

and hardware used for the games -

AlphaZero and the previous AlphaGo Zero used a single machine with 4 TPUs Stockfish and Elmo played at their strongest skill level using 64 threads and a hash size of 1GB.

So, AlphaZero used special hardware developed by Google. It used specialized Tensor Processor Units (TPUs) rather than general Central Processing Units (CPUs) as are available commercially.

This is how Wikipedia describes the second generation TPUs they used -

The second generation TPU was announced in May 2017. Google stated the first generation TPU design was memory bandwidth limited, and using 16 GB of High Bandwidth Memory in the second generation design increased bandwidth to 600 GB/s and performance to 45 TFLOPS. The TPUs are then arranged into 4-chip 180 TFLOPS modules

They used 4 TPUs for the games, so a processing power of 180 TFLOPS. Note TFLOPS = 1000 billion floating point operations per second.

For comparison Intel's latest most powerful chip is the Core i9 Extreme Edition processor which clocks in at 1 TFLOP. A top of the line I7 that you would find in a gaming machine would typically be about 100 GFLOPs (i.e. one tenth of a TFLOP).

I think it's fair to say that AlphaZero was using an 800 pound gorilla of a hardware configuration compared to Stockfishes mouse.

  • 1
    FLOPS stands for floating point operations per second. Floating point arithmetic is probably not used at all in the core algorithmes of Stockfish and AlphaZero. So the number of FLOPS is not really a meaningfull measure of processor speed relevant to the chess engine. – René Pijl Dec 8 '17 at 18:47
  • 7
    On the contrary, I believe neural nets use floating point arithmetic quite intensively. (But of course your remark makes perfect sense and applies to Stockfish.) – Federico Poloni Dec 8 '17 at 21:43
  • 2
    A far more apt comparison would be with a GPU; the NVIDIA Tesla V100 can do 120 TFLOPS, up from the previous generation (P100) which only did about 20. CPUs are optimized very differently from GPUs that do sheer volumes of numbers. – Nick T Dec 8 '17 at 21:53

I think it's best if I elaborate on your second point with an example move in the game 1 between AlphaZero and Stockfish which also served to satisfy my curiosity today.

the time limit of 1 min/move (How would this disadvantage Stockfish?)

Stockfish's performance is dependent upon both the time limit and the hardware configuration, so just think of when someone doubles the CPU threads, then Stockfish needs less time (not necessarily half) to find the solution than it would with the first configuration.

On the first report that was posted on Chess.com someone claimed that Stockfish was not playing optimally because he couldn't reproduce the same results using the same Stockfish on his computer. He said that on the position below (game 1 - move 11) Stockfish played Kg1-h1 (moved its king) which made no sense at all. On the other hand, stockfish on his computer showed a more developing move like Be3 (move the dark square bishop), lets look at the position:

Game 1 on move 11

Yes, it was a passive move and it seems that Stockfish should have played a more developing move. But he was wrong. Why? Because he ran Stockfish for 15 seconds, and if he had run it for an hour he would have gotten Kg1-h1 as the best move in that position. Stockfish changes it's decision when it analyses all the possible moves in more depth. Here's what I originally said in my reply:

I ran the latest stockfish on the position (at move 11):

  • At first, It gives b4 as the optimal move when the engine is running for about a minute. After that, it decides Be3 is better.
  • But after 5 minutes on my hardware that runs on 1,400k nodes/s it will decide to go with Kh1 as the optimal move.

  • In the paper, it is said that stockfish calculates 70,000k positions per second and is run for 1 minute per move, that's about 50 times my hardware, so I'll let mine run for 50 minutes... Kg1-h1 is still the choice for Stockfish.

Time limit is the key

In the above case, it probably didn't matter much if Stockfish ran for twice the time because the decision would have been the same, but on the next move it definitely would:

enter image description here

In this position, Stockfish chose to move the pawn on the left side (a4-a5). Let's say I have a computer that runs the Stockfish engine at a speed of 1,400k nodes per second, that's about 50 times lower than the Stockfish in the real game (In the paper, it says 70,000k n/s). So I can simulate the game if I run it for 50 minutes at each move. Okay.

I ran Stockfish analysis on the above position and I got the following results:

  • Stockfish started out suggesting some moves, but after 6 minutes on my computer (corresponds to 7.2 seconds on the Stockfish in the real game) it preferred a4-a5 just as the game went.

That's good, but I kept it running for a complete 50 minutes in order to reach the computations of the Stockfish in the game that was allowed 1 minute:

The sad truth is that I believe Stockfish lost all its games because of the time limit. Stockfish gets a more in-depth search and evaluation as the time passes and in the game it wasn't allowed to use an opening book which makes it consider many moves in shallow depths. Note that in the actual game a4-a5 was played which shows that (assuming it could evaluate 70 million positions per second) the Stockfish in the game didn't spend more than 21.6 seconds on the move. Otherwise, it would have changed its decision to those three other moves in the actual game. The reason for this is still unclear to me since my Stockfish was also consuming less memory (about ~130MB of RAM compared to the 1GB mentioned in the original paper, assuming all of it goes to hash tables).


Conclusion

The hardware that ran Stockfish, as I pointed out, was at the very best 18 times faster than mine (Update: on a single core) based on the move I analyzed. I'm not sure if AlphaZero could really make use of such hardware to train its networks in 4 hours, I can only assume it's too low for a game like chess. Besides, AlphaZero spent those hours on learning which also includes building solid openings (and as the paper points out, preferences over certain openings). On the other hand, Stockfish was handicapped on openings, and it did not evaluate 70 million positions per second for 60 seconds on each move.

As a final note, all the things I said were based on my assumptions. Of course, the outcome of AlphaZero and the games were super interesting to me. However, I would have loved to see a game where the Stockfish play was just like what I get on my computer, too. That is, more time and an opening book allowed. It's also easy to get the outputs of Stockfish analysis on every move, and I wish they release it in order to show how well it performed.

  • 1
    Regarding time limit, Figure 2 in AlphaZero paper shows opposite: Stockfish is better in lower budget, but scales worse, when more power is available. arxiv.org/pdf/1712.01815.pdf – old-ufo Dec 9 '17 at 22:40
  • 1
    @old-ufo Thanks for pointing that out. As I said, the performance of Stockfish (and AlphaZero) is dependent on the hardware as well as the time limit. For Instance, if we gave Stockfish more hardware (and AlphaZero less) and regenerated that figure, its line could be transformed such that it always stays above the AlphaZero line. So I guess that's a good point in comparing the hardware for both systems which also answers the main question. – AbdolrezaT Dec 10 '17 at 12:20

One of the original authors of Stockfish answers the specific complaints you mentioned here:

Meanwhile Chess.com also received a lengthy comment from one of the original Stockfish authors, Tord Romstad, which we'll give in full:

The match results by themselves are not particularly meaningful because of the rather strange choice of time controls and Stockfish parameter settings: The games were played at a fixed time of 1 minute/move, which means that Stockfish has no use of its time management heuristics (lot of effort has been put into making Stockfish identify critical points in the game and decide when to spend some extra time on a move; at a fixed time per move, the strength will suffer significantly). The version of Stockfish used is one year old, was playing with far more search threads than has ever received any significant amount of testing, and had way too small hash tables for the number of threads. I believe the percentage of draws would have been much higher in a match with more normal conditions.

On the other hand, there is no doubt that AlphaZero could have played better if more work had been put into the project (although the "4 hours of learning" mentioned in the paper is highly misleading when you take into account the massive hardware resources used during those 4 hours). But in any case, Stockfish vs AlphaZero is very much a comparison of apples to orangutans. One is a conventional chess program running on ordinary computers, the other uses fundamentally different techniques and is running on custom designed hardware that is not available for purchase (and would be way out of the budget of ordinary users if it were).

From another perspective, the apples vs orangutans angle is the most exciting thing about this: We now have two extremely different (both on the hardware and the software side) man-made entities that both display super-human chess playing abilities. That's much more interesting than yet another chess program that does the same thing as existing chess programs, just a little better. Furthermore, the adaptability of the AlphaZero approach to new domains opens exciting possibilities for the future.

For chess players using computer chess programs as a tool, this breakthrough is unlikely to have a great impact, at least in the short term, because of the lack of suitable hardware for affordable prices.

For chess engine programmers -- and for programmers in many other interesting domains -- the emergence of machine learning techniques that require massive hardware resources in order to be effective is a little disheartening. In a few years, it is quite possible that an AlphaZero like chess program can be made to run on ordinary computers, but the hardware resources required to create them will still be way beyond the budget of hobbyists or average sized companies. It is possible that an open source project with a large distributed network of computers run by volunteers could work, but the days of hundreds of unique chess engines, each with their own individual quirks and personalities, will be gone.

Source: https://www.chess.com/news/view/alphazero-reactions-from-top-gms-stockfish-author

Visit Talkchess Forum to know more, there is where you will find some 3000 programmers. This was all a scam. Alpha played on 30 times bigger hardware than SF, 4TPUs vs 64 cores. 4TPUs is around 1000 cores or even more. Alpha had simulated opening book, trained on countless top GM winning games. SF had very little hash. TC was fixed at 1 minute per move, which is again detrimental to SF, which has advanced time management. TPUs lack the SMP inefficiencies with more cores, so the hardware advantage was even bigger. Etc, etc., so basically, this was just a huge publicity stunt on the part of Google. Currently, Alpha is around 2800 on single core, so 400 elos below SF, and will not advanced much in the future, as, from now on, it will need advanced evaluation it will not be able to discover. Concerning the 4-hours issue, well, LOL, this was 48 hours ago, so now Alpha is at 5000 elo? Come on.

  • 4
    You seem to believe that AlphaZero does the same thing as Stockfish, only 1000 times faster because it used 1000 times stronger hardware. This is not true at all. It uses a very different approach and that approach is very resource intensive. In fact, during the match AlphaZero was evaluating 80 thousand positions per second while Stockfish was clocking at 70 million positions per second. Now tell me that AlphaZero won because of a stronger hardware. Of course on 64 CPU it would be slower and who knows how it would play but the point is that AlphaZero does it better, albeit at higher cost. – IA Petr Harasimovic Dec 9 '17 at 15:28
  • 3
    SF hardware costs less than $10k, Alpha one more than $250k. Draw the conclusions yourself. Nps are meaningless, and every chess programmer knows that. You can do all kinds of tricks so that nps get lower, but that does not mean you will play stronger. I would like to see it implement that approach on SF hardware and SF its on Alpha hardware. Guess the result? +85 -0 =15 for SF. If they are so great, let them publish their code. – Lyudmil Tsvetkov Dec 10 '17 at 13:29
  • 2
    "Alpha had simulated opening book, trained on countless top GM winning games." Simulated opening book, yes, but it trained on GM games? Do you have a source for this? My understanding was that Alpha was entirely bootstrapped. – Akavall Dec 10 '17 at 18:39

Running on comparable hardware would be required if Google's end goal was to build a superior chess engine, but this exercise wasn't really about chess. Chess is just a convenient way to demonstrate the AI's ability to learn complicated tasks from scratch. If it can perform well against some vaguely reasonable configuration of Stockfish, it's checked the box.

I predict the Google team will not spend much more effort on chess; instead, they will move on to other problems that AI has never been able to accomplish.

  • I've given +1 because I have the same feeling. – SmallChess Dec 19 '17 at 0:40
  • Sounds probable, though I doubt they would have published it, if Alphazero had lost by only a small margin (meaning that it would still be of comparable strength to stockfish). – user1583209 Dec 19 '17 at 21:13
  • @user1583209 They probably ran it lots of times to figure out the minimum amount of learning time it needed to crush Stockfish. Then they did a final run and published those results. – T Scherer Dec 20 '17 at 0:32

Stockfish is constrained to CPUs so it will never be able to scale to the level that GPUs are able.

Gor matrix calculations GPUs scale with the n, while CPUs scale with n3, these tensor cores are further optimized so it's likely even better performance as you scale.

New contributor
Famous Jameis is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

First paragraph more detail, second short and simple answer third paragraph my opinions on the situation

With AlphaZero the hardware has 0 effect on the strength of its play. It may take longer but not because it’s thinking. It’s a neural network, which means you feed it info in a vector (a single column table) it does simple math thru a giant tensor (a 3 or more dimensional table) then it spits out the answer. Stockfish needs time to be good because it checks possible positions to see if a move is beneficial, so the longer it looks at the problem the more position/moves it can check.

There isn’t really a comparable hardware setup. Because they have different needs, Stockfish needs to analyze more positions while AlphaZero just needs to make a move. And people are upset because AlphaZero's computer is technically much more powerful and they think they should be equal in that regard. But, AlphaZero doesn’t need that supercomputer after training.

In my opinion it doesn’t matter what they give to either side, unless Stockfish has an unreasonable amount of time it will probably tie a few more games but in general a similar effect will happen. This is why I think this, Stockfish initially evaluates with pieces and their values while, Alpha played (probably) millions of games to realize what is important strategically. Which is why Alpha sacs a lot more than Stockfish would ever, but gains huge positional advantages.

  • That's just wrong. AlphaZero does tree search. More hardware makes it stronger. And it needs a ton of hardware to play better than stockfish. – BlindKungFuMaster Dec 13 '17 at 9:16
  • Nothing in this post is correct... – SmallChess Dec 13 '17 at 10:51
  • This system is based on 3 neuralnets and partial Monroe Carlo tree search, so you are correct about it using tree search. During play it uses 2 neural network principles developed in alphago of value and policy. Go is a game that can’t be done via computational power because it’s astronomically more complex than chess. So if I believe that it needs more computing power than a brute force algorithm then you are delusional. Or miss informed. The third neural net is used to try and excelerate the learning process by guessing the back propagation changes. Alphazeros power is in the nets not mcts. – Ezecal Dec 14 '17 at 14:33

protected by Phonon Sep 15 at 9:53

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.