Are there any papers that highlight the problems with rating systems?

Question

Ratings are a paradox. Mathematically they seem to dependent on other factors than ability. Yet people are obsessed with their rating as if it defines them as a person somehow.

Are their any papers that discuss the many problems that all current rating systems have along with their shortcomings and the ASSUMPTIONS that were used in deciding how to create a rating system.

Are there any statistical studies showing the distribution of wins/draws based on rating differences that have been validated and show whether the assumptions used in FIDE tables and other places are accurate?

OTOH Ratings are useful and 'close enough for government work' But they fall far short of what many people believe about them.

Answer 1

It's a bit old, but this paper from 1995 by Mark Glickman (creator of the Glicko rating system) discusses some of the rating systems in use at the time, including the ones used by the USCF, FIDE, and the PCA. In it, several issues are mentioned, such as rating inflation/deflation, regional variations in ratings, variations in time controls, and the advantage of White. There's some other interesting stuff in there, like how natural ratings variation means that a stronger player will have a less than predicted actual performance against weaker players.

Not everything in it is still accurate. For example, at one point it states that the USCF rates events in the order they are received rather than the order in which they were played, but currently the USCF will re-rate the more recent event if an older event is submitted to it, so that is no longer a problem with that system. It also states that FIDE has a minimum rating of 2000, and that is no longer the case. The paper also obviously doesn't discuss any of the more recent systems. But it may be a good starting point.