In light of the ongoing TCEC tournament, I was thinking, how do they come up with the ratings of these computers? Presumably, a 2000 rated chess engine should expect a score of about .5 against a human rated 2000. This perhaps makes it easy to verify if chess engines are really approximately the rating they say they are. However, this is problematic for higher rated programs like komodo, who is supposedly around 3250. Obviously, no humans play at this level. Furthermore, even 2800 rated computers do not play against humans(I doubt magnus plays against computers to help developers define their rating). It is even suspicious if engines play against grandmasters to determine their strength. So, I suspect that these engine ratings are determined by engines playing opponents of similar strength a long time ago, and most ratings come from engine vs engine games these days. But if this was true, I wouldn't expect that engine ratings are accurate in comparison to the human elo scale. Is there a way that they check that engine ratings are accurate?
Computer chess engine ratings are not related to the FIDE rating list.
For example, the
MicroMax engine has around 1950 rating on CCRL:
I distribute an iOS app (ChessMini) for this engine. I wrote on the app description page:
The engine has a rating of Elo 2000 on the Computer Chess Rating Lists and is stronger than 70% of the human chess players.
This is true only if the CCRL chess rating is a close approximation to the FIDE rating list.
Unfortunately, I've received several users complained that they were able to beat the app convincingly despite their FIDE ratings are much less than 1900. In fact, I have a FIDE rating around 2000 myself but the MicroMax engine has never beaten me. I think it's "real" rating on the FIDE scale is 1300-1400.
Forget about the human rating list, just look at the relative ranking.
Computer chess ratings are extraordinary accurate (but no relationship to FIDE) because the sample size is only bounded by your processing power. You start an engine tournament, leave the computer running and goto bed... It's 24 hours chess like TCEC. You play as many engine matches as your computer can afford.
Computer engines get a rating by playing other engines. No human intervention. No grandmaster. Anybody with a laptop can do that:
That's it, it's so simple! You'll get estimated ratings for your engines.