How well does Elo rating estimate a player's strength?

Question

Proposal: strength is the real and rating is the model that estimates it.

Question: How well does Elo rating estimate a player's strength?

Bonus question: What happens if you take the player's rating curve into account?

Answer 1

The Elo rating is a statistical prediction of your results against players with known rating when they play each other in the long term. Some examples:

• Two players with the same rating will score the same amount of victories when playing together.

• Two players with a 200 Elo difference, means the stronger player will win 3 out of 4 games (and lose the other).

• Two players with a 400 Elo difference, means the stronger player will win 10 out of 11 times (and the weakest winning the other one).

• And so on, a difference beyond 700 is not accurately measurable (basically the system predicts 100% score of the stronger player).

Now, in practice, many other factors come into play (ie. the stronger player being bored or overconfident, playing a system in which the weakest players has in-depth knowledge, being in shape, etc.)

For more details, see

https://thechessworld.com/articles/general-information/deeper-look-at-elo-chess-rating-system/

http://en.wikipedia.org/wiki/Elo_rating

Answer 2

It's not clear to me whether you are asking a philosophical or a quantitative question.

My answer to the philosophical question is that the Elo rating is predictive; that is, it is designed to make accurate predictions about the outcomes of chess games. To that extent, it measures strength by estimating the likelihood that a player will win against another player. It does this by using the player's past results, and it does the best it can given that information.

The answer to the quantitative question is that the "expected value" of a game between two players is correlated with the difference in their Elo ratings. For example if player A's Elo rating is 200 higher than that of player B, then player A would be expected to go about 76-24 over the course of a 100-game match.

Note: The question has changed since I answered it, so my answer makes less sense now. The title of the question used to be Does the chess rating represent a player's strength?, and the question was "I notice that somehow, we chess players associate the chess rating with strength. But is it a measure of strength? To what extent?" Please take this answer as a response to that question, not the one it has been changed to.

Answer 3

"Normally" (pun intended) the Elo rating does indicate a player's average strength (you tagged this post with Elo, so I am considering Elo rating).

Elo rating assumes that a player's performance conforms to what is called the Normal Distribution. Thus, if your rating is 2700, then that's your mean or average level of performance. However, that does not give an indication of how far varied your performance can be in relation to your mean rating.

For instance, if your standard deviation is low, then it means that your performance is very consistent around 2700 (for example, you beat most guys below 2600, hardly ever beat 2800+, and usually draw with players between that range). If your standard deviation is high, then it means that sometimes, you beat even Magnus Carlsen (2881 as of today), but at times you find yourself losing to a low rated Grandmaster or even an International Master.

Answer 4

The Elo rating system doesn't estimate a player's strength as well as other systems. I've read a while ago about a contest between rating systems, I'll describe it by quoting the article: competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games.

The conclusion:

The Elo benchmark was outperformed within 24 hours

On a sidenote, I can't see the charts anymore, if anyone finds them, you're welcome to contribute them. From what I remember there were no less than 60 rating systems which outperformed Elo.

Answer 5

The weakness of the Elo rating system is that it depends on scoring, that is the assignment of win=1, loss=0, and draw=1/2. Scoring causes some information about chess game outcomes to be lost. It would be more accurate to measure chess playing strength in two dimensions. Please see: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2477868

Answer 6

An Elo rating most emphatically is NOT a measure of a player's strength, nor was it intended to be. A player might, for all kinds of reasons, have a lower Elo rating than someone that is in reality weaker. For example a player that plays mostly in team competitions might accept a draw in a winning position, or risk losing by playing for a win in a drawn position, with bad effects on their rating.

Another example: One player in SWISS tournaments might play objectively to get the "correct" number of points that their play deserves. Another might take great risks on the grounds that 3.5 or 4 out of 6 finishes nowhere whereas 5 or more is usually in the money, and worth the risk of losing from an easily drawn position. The first player would probably earn a higher rating, but the second would win more tournaments! (and might actually be stronger).

The maths behind Elo ratings guarantees that rating points are gained more slowly than is warranted by current form and also ensures that a few bad results do not reduce a rating by a huge amount. In effect games played years earlier when a player was much weaker (or stronger) affect the rate at which the rating can change. This aspect of controlling the rate of change is further fudged by choice of k-factor.

A further problem is that Elo ratings only apply properly to players that are contemporary. They do not give meaningful comparisons of strength for players active in different eras. Of course that has not stopped people from making such comparisons and concluding (for example) that Lasker and ZUuckertort would hold their own against today's GM's that former world-champion Petrosian was no stronger than any modern-day top-50 super-GM, or that Fischer was significantly weaker than Kasparov and Carlsen. Those conclusions may or may not be true, but it is just wrong to use Elo ratings (retrospectively calculated as necessary) to arrive at them.

What Elo ratings are designed for is to allow the probability of encounters between players to be estimated. They do that quite well for players that have stable ratings that have not changed much over a few years. They do a poor job when one or both of the players is either improving or declining.

They have been adopted as a proxy for chess strength, but are far from perfect. The right way to assess strength is to study a players games and make a judgement, but to do that you have to be very strong yourself, and it is hard work.

As in so many other areas of life and human endeavour it is much easier to pretend that a simple number based on results tells us what we'd like to know ... even when it doesn't.

Answer 7

One interpretation of the question is "What is the standard error of Elo?" That is, if Elo is a statistic that measures some true latent ability, what is the standard deviation of that statistic? If, for example, you've just played 50 tournament games, and your Elo is calculated as 1700, is it really 1700, or might it actually be 1710? Or 1750? Might it be 1600?

I have analyzed this question in a blog post, the gist of which is as follows: Suppose two players play each other repeatedly. The difference in true abilities is "Delta A", where we assume the probability of a win by the first player is the Elo logistic function in Delta A. Then, after many games between them, the difference in their computed Elos will be Delta r which equals Delta a plus some error, epsilon. That error term, epsilon, approximately follows an autoregressive process, and is asymptotically unbiased. The standard error depends on the "k factor" used in updating Elo, and on the probability of a tie (which is typically higher for better players, when equally matched). Here is a plot of the empirical (dots) and theoretical standard error versus the k factor, for different probabilities of a tie: The upshot is that differences in Elo will have standard error around 20 or more.

As others have noted, the Elo rating has some deficiencies. However, it is fairly simple to describe and compute, which was necessary at the time of its introduction.