I think it's best if I elaborate on your second point with an example move in the game 1 between AlphaZero and Stockfish which also served to satisfy my curiosity today.
the time limit of 1 min/move (How would this disadvantage Stockfish?)
Stockfish's performance is dependent upon both the time limit and the hardware configuration, so just think of when someone doubles the CPU threads, then Stockfish needs less time (not necessarily half) to find the solution than it would with the first configuration.
On the first report that was posted on Chess.com someone claimed that Stockfish was not playing optimally because he couldn't reproduce the same results using the same Stockfish on his computer. He said that on the position below (game 1 - move 11) Stockfish played Kg1-h1 (moved its king) which made no sense at all. On the other hand, stockfish on his computer showed a more developing move like Be3 (move the dark square bishop), lets look at the position:
Yes, it was a passive move and it seems that Stockfish should have played a more developing move. But he was wrong. Why? Because he ran Stockfish for 15 seconds, and if he had run it for an hour he would have gotten Kg1-h1 as the best move in that position. Stockfish changes it's decision when it analyses all the possible moves in more depth. Here's what I originally said in my reply:
I ran the latest stockfish on the position (at move 11):
- At first, It gives b4 as the optimal move when the engine is running for about a minute. After that, it decides Be3 is better.
But after 5 minutes on my hardware that runs on 1,400k nodes/s it will decide to go with Kh1 as the optimal move.
In the paper, it is said that stockfish calculates 70,000k positions per second and is run for 1 minute per move, that's about 50 times my hardware, so I'll let mine run for 50 minutes... Kg1-h1 is still the choice for Stockfish.
Time limit is the key
In the above case, it probably didn't matter much if Stockfish ran for twice the time because the decision would have been the same, but on the next move it definitely would:
In this position, Stockfish chose to move the pawn on the left side (a4-a5). Let's say I have a computer that runs the Stockfish engine at a speed of 1,400k nodes per second, that's about 50 times lower than the Stockfish in the real game (In the paper, it says 70,000k n/s). So I can simulate the game if I run it for 50 minutes at each move. Okay.
I ran Stockfish analysis on the above position and I got the following results:
- Stockfish started out suggesting some moves, but after 6 minutes on my computer (corresponds to 7.2 seconds on the Stockfish in the real game) it preferred a4-a5 just as the game went.
That's good, but I kept it running for a complete 50 minutes in order to reach the computations of the Stockfish in the game that was allowed 1 minute:
The sad truth is that I believe Stockfish lost all its games because of the time limit. Stockfish gets a more in-depth search and evaluation as the time passes and in the game it wasn't allowed to use an opening book which makes it consider many moves in shallow depths. Note that in the actual game a4-a5 was played which shows that (assuming it could evaluate 70 million positions per second) the Stockfish in the game didn't spend more than 21.6 seconds on the move. Otherwise, it would have changed its decision to those three other moves in the actual game. The reason for this is still unclear to me since my Stockfish was also consuming less memory (about ~130MB of RAM compared to the 1GB mentioned in the original paper, assuming all of it goes to hash tables).
The hardware that ran Stockfish, as I pointed out, was at the very best 18 times faster than mine (Update: on a single core) based on the move I analyzed. I'm not sure if AlphaZero could really make use of such hardware to train its networks in 4 hours, I can only assume it's too low for a game like chess. Besides, AlphaZero spent those hours on learning which also includes building solid openings (and as the paper points out, preferences over certain openings). On the other hand, Stockfish was handicapped on openings, and it did not evaluate 70 million positions per second for 60 seconds on each move.
As a final note, all the things I said were based on my assumptions. Of course, the outcome of AlphaZero and the games were super interesting to me. However, I would have loved to see a game where the Stockfish play was just like what I get on my computer, too. That is, more time and an opening book allowed. It's also easy to get the outputs of Stockfish analysis on every move, and I wish they release it in order to show how well it performed.