2

AlphaZero has defeated Stockfish with all novelties and brilliant strategic sacrifices. But some people were disputing its win, because Stockfish was running on weak hardware (1 GB RAM), not access to opening book, not having free time control, maybe not have endgame table base, etc.

This setting by DeepMind is understandable from a scientific perspective though. It is to test intuitive/strategic play by deep learning technique vs. counting/tactical play by brute force search. However, this leaves chess lovers with many questions like what is the true strength of AlphaZero? What is the true advantage of strategic/positional sacrifice play? Were Morphy, Tal, and others correct and Stockfish wrong? Or full version Stockfish is still the best?

One work around is to replicate the Stockfish version that played AlphaZero, to see how strong it is compared to the full version and other engine and human. Thus we could infer the Elo rating of AlphaZero and answer most questions.

So, did anyone test it? Is anyone interested in testing it? Maybe Stockfish developers, because Stockfish's pride is at stake? Or is there other way to approach the problem?

Thanks

Edit:

  • Note that Stockfish had good 64 threads CPU and could evaluate 70 million positions/second.
  • The Stockfish Elo rating provided in the paper is the full version, thus this replication is needed.
  • The paper.
Improve this question
5
  • 1
    I assume any down vote has a reason, but please share it.
    – THN
    Mar 17, 2018 at 10:11
  • 2
    You are not going to find a (human) Elo rating for stockfish. The best you can hope for is a computer Elo rating determined from many games between various computer engines. Even then, I am not sure what this would tell about the rating of AlphaZero, because AlphaZero only played against some version of stockfish and not against any of the other computer engines in the pool.
    – user1583209
    Mar 17, 2018 at 12:35
  • @user1583209 I think AlphaZero is 100 Elo higher than that version of Stockfish. If we know the Elo of that version, we would have some more reliable evidence when talking about the strength of AlphaZero. Most reviews now are wildly compliments or disputes.
    – THN
    Mar 17, 2018 at 13:17
  • 1
    @THN You won't find any reliable human rating for Stockfish.
    – SmallChess
    Mar 17, 2018 at 13:55
  • 1
    The suggestion is made that Stockfish cannot reliably be rated against humans. I read suggestions like this often (often coupled with suggestions—equally false—that online and federation ratings cannot be correlated). I believe the suggestion to be a canard. As far as a I know, nothing statistically prevents a reasonably accurate rating of computers on an extrapolated human scale.
    – thb
    Jun 23, 2018 at 19:45

3 Answers 3

Reset to default
4

The Stockfish version that played agaist AlphaZero was significantly compromised in several different aspects:

  1. There was no opening book available
  2. Time control was 1 minute/move, which throws out of the window various optimizations re: using more time thinking about difficult moves
  3. Stockfish had threads=64 set, but the actual hardware had only 8 CPU cores, that effectively slowed things down quite a lot
  4. The hardware was quite different -- an average $1k PC (Stockfish) against a bunch of specialized hardware that might cost millions and is not accessible to anyone
  5. There are more, but these are main points.

Answering your question, there's no way to measure how much rating Stockfish had in a very handicapped conditions it had to run.

Nonetheless, winning over Stockfish is quite an achievement, no matter how handicapped it was and how much was the difference in hardware cost.

UPDATE: in the recent article "AlphaZero: Shedding new light on the grand games of chess, shogi and Go" the rating for Stockfish was drawn as a line at 3500. Besides that, the article provides the new games played with a reasonable time control for SF as well as opening books.

Improve this answer
4
  • "3." is new news to me, do you have source for it? Also, I want to avoid any dispute between AZ and SF, because there are too many disputes on the net already. Let's focus the question on fact-based comparison. The strength of compromised SF could be determined empirically if (1) someone have the equivalent hardware, (2) they replicate the SF settings, (3) they let it play against other engines and compute the Elo ratings. (1) may not necessary if one let SF run longer for each move until it reach 70*60 mil. positions.
    – THN
    Jun 24, 2018 at 3:49
  • 1
    @THN (3) was discussed by SF author in the interview after SF/AZ game, he complained about other things that put SF at a great disadvantage, but I just don't remember 'em all. (1) is necessary to save time, SF is optimized not to spend too much time in debut because it's supposed to have an opening book for that. the rest -- I don't have the HW nor the exact settings and don't have time to set up computer matches. My estimation is something around 3200 3400, but this is quite useless, taking into account the fact there are no humans playing above 3k and all those numbers have no real meaning.
    – lenik
    Jun 24, 2018 at 4:22
  • Thanks for the info, please update links to source if you found. From the paper I still think they used 64 cores CPU. Elo ratings have very nice interpretation in Bayesian approach, high Elo can also be computed (extrapolated) with high probability credible interval. Not many people can do experiments, so it is just a call for contribution from those who have hardware.
    – THN
    Jun 24, 2018 at 5:33
  • @THN a quick google search came out with this: reddit.com/r/chess/comments/7igro1/… I'll post the link where 8 CPU cores are mentioned later, if I find it.
    – lenik
    Jun 24, 2018 at 6:02
3

AlphaZero vs. Stockfish 8, 1000-game match as in the latest paper (with Stockfish operating at full power) yielded a score of +155 -6 =839. Using this calculator gives an elo difference of 52.

Stockfish 8's elo rating on computer chess rating lists is about 3378, giving AlphaZero a rating of about 3430.

Disclaimer: these rating lists are generated with hundreds of thousands of games between all the engines. With only 1000 games against a single engine, this 3430 rating is unlikely to be accurate. AlphaZero is certainly one of the stronger engines out there, but it's not clear exactly how strong.

Improve this answer
2

"Stockfish was running on weak hardware (1 GB RAM)...This setting by DeepMind is understandable from a scientific perspective..."

I'm a master scientist and this is NOT understandable scientifically. You have introduced an unneeded and unwanted variable by running Stockfish on weak hardware. Poor experimental design. Poor science.

Improve this answer
1
  • Controlled experiment is a common practice to isolate an effect of interest. On a side note, this should be in a comment of the question, because it's not an answer.
    – THN
    Dec 5, 2018 at 0:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.