In light of the ongoing TCEC tournament, I was thinking, how do they come up with the ratings of these computers? Presumably, a 2000 rated chess engine should expect a score of about .5 against a human rated 2000. This perhaps makes it easy to verify if chess engines are really approximately the rating they say they are. However, this is problematic for higher rated programs like komodo, who is supposedly around 3250. Obviously, no humans play at this level. Furthermore, even 2800 rated computers do not play against humans(I doubt magnus plays against computers to help developers define their rating). It is even suspicious if engines play against grandmasters to determine their strength. So, I suspect that these engine ratings are determined by engines playing opponents of similar strength a long time ago, and most ratings come from engine vs engine games these days. But if this was true, I wouldn't expect that engine ratings are accurate in comparison to the human elo scale. Is there a way that they check that engine ratings are accurate?
-
1If you can get a reliable rated human, play a match vs a weak computer, to "calibrate" the computer elo, then later you may use that engine vs other engine 300 points higher to calibrate that one, and go staircase like that, 300 elo points each time to make it statistically easy (so that not all are wins, because then how do you know how much points better it is) i think it would be feasible to at least esstimate elo of strong engines.– SantropedroCommented Oct 28, 2017 at 6:08
1 Answer
Computer chess engine ratings are not related to the FIDE rating list.
For example, the MicroMax engine has around 1950 rating on CCRL:
I distribute an iOS app (ChessMini) for this engine. I wrote on the app description page:
The engine has a rating of Elo 2000 on the Computer Chess Rating Lists and is stronger than 70% of the human chess players.
This is true only if the CCRL chess rating is a close approximation to the FIDE rating list.
Unfortunately, I've received several users complained that they were able to beat the app convincingly despite their FIDE ratings are much less than 1900. In fact, I have a FIDE rating around 2000 myself but the MicroMax engine has never beaten me. I think it's "real" rating on the FIDE scale is 1300-1400.
Forget about the human rating list, just look at the relative ranking.
Computer chess ratings are extraordinary accurate (but no relationship to FIDE) because the sample size is only bounded by your processing power. You start an engine tournament, leave the computer running and goto bed... It's 24 hours chess like TCEC. You play as many engine matches as your computer can afford.
Computer engines get a rating by playing other engines. No human intervention. No grandmaster. Anybody with a laptop can do that:
That's it, it's so simple! You'll get estimated ratings for your engines.
-
1Still there should be some approximate relation between the two lists no? (It's hard to find such a relation with few games played between humans and computers, but there should be one.)– TMMCommented Oct 20, 2017 at 13:53
-
@TMM Maybe. If there is one and it can be proven statistically (something like correlation), please attempt your answer. Commented Oct 20, 2017 at 13:54
-
1Your answer offers an interesting anecdote and is valuable to that extent, but does not seem authoritative in the specific matter OP raises. Your answer is your answer, so I do not think that you should change it unless you just want to; but I nevertheless believe that the answer would have been more accurate had it begun, "I do not know the answer to your question, but here is an illuminating anecdote." In my view, therefore, the question is still open and still wants a proper answer.– thbCommented May 19, 2019 at 12:16