I understand that AlphaZero has to use a different kind of hardware than regular Stockfish. I would expect that the hardware has a large effect on engine strength. That's why I wonder whether there have been any attempts made to provide comparable hardware to both. Also what would "comparable" mean here?
Specifically I read that people complain about:
That's why I wonder whether there have been any attempts made to provide comparable hardware to both.
This is Google you're talking about! So the answer is obviously "No".
From the original paper hardware used for initialising and training -
Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation TPUs to train the neural networks
and hardware used for the games -
AlphaZero and the previous AlphaGo Zero used a single machine with 4 TPUs Stockfish and Elmo played at their strongest skill level using 64 threads and a hash size of 1GB.
So, AlphaZero used special hardware developed by Google. It used specialized Tensor Processor Units (TPUs) rather than general Central Processing Units (CPUs) as are available commercially.
This is how Wikipedia describes the second generation TPUs they used -
The second generation TPU was announced in May 2017. Google stated the first generation TPU design was memory bandwidth limited, and using 16 GB of High Bandwidth Memory in the second generation design increased bandwidth to 600 GB/s and performance to 45 TFLOPS. The TPUs are then arranged into 4-chip 180 TFLOPS modules
They used 4 TPUs for the games, so a processing power of 180 TFLOPS. Note TFLOPS = 1000 billion floating point operations per second.
For comparison Intel's latest most powerful chip is the Core i9 Extreme Edition processor which clocks in at 1 TFLOP. A top of the line I7 that you would find in a gaming machine would typically be about 100 GFLOPs (i.e. one tenth of a TFLOP).
I think it's fair to say that AlphaZero was using an 800 pound gorilla of a hardware configuration compared to Stockfishes mouse.
I think it's best if I elaborate on your second point with an example move in the game 1 between AlphaZero and Stockfish which also served to satisfy my curiosity today.
the time limit of 1 min/move (How would this disadvantage Stockfish?)
Stockfish's performance is dependent upon both the time limit and the hardware configuration, so just think of when someone doubles the CPU threads, then Stockfish needs less time (not necessarily half) to find the solution than it would with the first configuration.
On the first report that was posted on Chess.com someone claimed that Stockfish was not playing optimally because he couldn't reproduce the same results using the same Stockfish on his computer. He said that on the position below (game 1 - move 11) Stockfish played Kg1-h1 (moved its king) which made no sense at all. On the other hand, stockfish on his computer showed a more developing move like Be3 (move the dark square bishop), lets look at the position:
Yes, it was a passive move and it seems that Stockfish should have played a more developing move. But he was wrong. Why? Because he ran Stockfish for 15 seconds, and if he had run it for an hour he would have gotten Kg1-h1 as the best move in that position. Stockfish changes it's decision when it analyses all the possible moves in more depth. Here's what I originally said in my reply:
I ran the latest stockfish on the position (at move 11):
But after 5 minutes on my hardware that runs on 1,400k nodes/s it will decide to go with Kh1 as the optimal move.
In the paper, it is said that stockfish calculates 70,000k positions per second and is run for 1 minute per move, that's about 50 times my hardware, so I'll let mine run for 50 minutes... Kg1-h1 is still the choice for Stockfish.
In the above case, it probably didn't matter much if Stockfish ran for twice the time because the decision would have been the same, but on the next move it definitely would:
In this position, Stockfish chose to move the pawn on the left side (a4-a5). Let's say I have a computer that runs the Stockfish engine at a speed of 1,400k nodes per second, that's about 50 times lower than the Stockfish in the real game (In the paper, it says 70,000k n/s). So I can simulate the game if I run it for 50 minutes at each move. Okay.
I ran Stockfish analysis on the above position and I got the following results:
That's good, but I kept it running for a complete 50 minutes in order to reach the computations of the Stockfish in the game that was allowed 1 minute:
The sad truth is that I believe Stockfish lost all its games because of the time limit. Stockfish gets a more in-depth search and evaluation as the time passes and in the game it wasn't allowed to use an opening book which makes it consider many moves in shallow depths. Note that in the actual game a4-a5 was played which shows that (assuming it could evaluate 70 million positions per second) the Stockfish in the game didn't spend more than 21.6 seconds on the move. Otherwise, it would have changed its decision to those three other moves in the actual game. The reason for this is still unclear to me since my Stockfish was also consuming less memory (about ~130MB of RAM compared to the 1GB mentioned in the original paper, assuming all of it goes to hash tables).
The hardware that ran Stockfish, as I pointed out, was at the very best 18 times faster than mine (Update: on a single core) based on the move I analyzed. I'm not sure if AlphaZero could really make use of such hardware to train its networks in 4 hours, I can only assume it's too low for a game like chess. Besides, AlphaZero spent those hours on learning which also includes building solid openings (and as the paper points out, preferences over certain openings). On the other hand, Stockfish was handicapped on openings, and it did not evaluate 70 million positions per second for 60 seconds on each move.
As a final note, all the things I said were based on my assumptions. Of course, the outcome of AlphaZero and the games were super interesting to me. However, I would have loved to see a game where the Stockfish play was just like what I get on my computer, too. That is, more time and an opening book allowed. It's also easy to get the outputs of Stockfish analysis on every move, and I wish they release it in order to show how well it performed.
One of the original authors of Stockfish answers the specific complaints you mentioned here:
Meanwhile Chess.com also received a lengthy comment from one of the original Stockfish authors, Tord Romstad, which we'll give in full:
The match results by themselves are not particularly meaningful because of the rather strange choice of time controls and Stockfish parameter settings: The games were played at a fixed time of 1 minute/move, which means that Stockfish has no use of its time management heuristics (lot of effort has been put into making Stockfish identify critical points in the game and decide when to spend some extra time on a move; at a fixed time per move, the strength will suffer significantly). The version of Stockfish used is one year old, was playing with far more search threads than has ever received any significant amount of testing, and had way too small hash tables for the number of threads. I believe the percentage of draws would have been much higher in a match with more normal conditions.
On the other hand, there is no doubt that AlphaZero could have played better if more work had been put into the project (although the "4 hours of learning" mentioned in the paper is highly misleading when you take into account the massive hardware resources used during those 4 hours). But in any case, Stockfish vs AlphaZero is very much a comparison of apples to orangutans. One is a conventional chess program running on ordinary computers, the other uses fundamentally different techniques and is running on custom designed hardware that is not available for purchase (and would be way out of the budget of ordinary users if it were).
From another perspective, the apples vs orangutans angle is the most exciting thing about this: We now have two extremely different (both on the hardware and the software side) man-made entities that both display super-human chess playing abilities. That's much more interesting than yet another chess program that does the same thing as existing chess programs, just a little better. Furthermore, the adaptability of the AlphaZero approach to new domains opens exciting possibilities for the future.
For chess players using computer chess programs as a tool, this breakthrough is unlikely to have a great impact, at least in the short term, because of the lack of suitable hardware for affordable prices.
For chess engine programmers -- and for programmers in many other interesting domains -- the emergence of machine learning techniques that require massive hardware resources in order to be effective is a little disheartening. In a few years, it is quite possible that an AlphaZero like chess program can be made to run on ordinary computers, but the hardware resources required to create them will still be way beyond the budget of hobbyists or average sized companies. It is possible that an open source project with a large distributed network of computers run by volunteers could work, but the days of hundreds of unique chess engines, each with their own individual quirks and personalities, will be gone.
Source: https://www.chess.com/news/view/alphazero-reactions-from-top-gms-stockfish-author
Running on comparable hardware would be required if Google's end goal was to build a superior chess engine, but this exercise wasn't really about chess. Chess is just a convenient way to demonstrate the AI's ability to learn complicated tasks from scratch. If it can perform well against some vaguely reasonable configuration of Stockfish, it's checked the box.
I predict the Google team will not spend much more effort on chess; instead, they will move on to other problems that AI has never been able to accomplish.
Visit Talkchess Forum to know more, there is where you will find some 3000 programmers. This was all a scam. Alpha played on 30 times bigger hardware than SF, 4TPUs vs 64 cores. 4TPUs is around 1000 cores or even more. Alpha had simulated opening book, trained on countless top GM winning games. SF had very little hash. TC was fixed at 1 minute per move, which is again detrimental to SF, which has advanced time management. TPUs lack the SMP inefficiencies with more cores, so the hardware advantage was even bigger. Etc, etc., so basically, this was just a huge publicity stunt on the part of Google. Currently, Alpha is around 2800 on single core, so 400 elos below SF, and will not advanced much in the future, as, from now on, it will need advanced evaluation it will not be able to discover. Concerning the 4-hours issue, well, LOL, this was 48 hours ago, so now Alpha is at 5000 elo? Come on.
Stockfish is constrained to CPUs so it will never be able to scale to the level that GPUs are able.
Gor matrix calculations GPUs scale with the n, while CPUs scale with n3, these tensor cores are further optimized so it's likely even better performance as you scale.
First paragraph more detail, second short and simple answer third paragraph my opinions on the situation
With AlphaZero the hardware has 0 effect on the strength of its play. It may take longer but not because it’s thinking. It’s a neural network, which means you feed it info in a vector (a single column table) it does simple math thru a giant tensor (a 3 or more dimensional table) then it spits out the answer. Stockfish needs time to be good because it checks possible positions to see if a move is beneficial, so the longer it looks at the problem the more position/moves it can check.
There isn’t really a comparable hardware setup. Because they have different needs, Stockfish needs to analyze more positions while AlphaZero just needs to make a move. And people are upset because AlphaZero's computer is technically much more powerful and they think they should be equal in that regard. But, AlphaZero doesn’t need that supercomputer after training.
In my opinion it doesn’t matter what they give to either side, unless Stockfish has an unreasonable amount of time it will probably tie a few more games but in general a similar effect will happen. This is why I think this, Stockfish initially evaluates with pieces and their values while, Alpha played (probably) millions of games to realize what is important strategically. Which is why Alpha sacs a lot more than Stockfish would ever, but gains huge positional advantages.
