# How well does Elo rating estimate a player's strength?

Edit (3 March 2014):

Proposal: strength is the real and rating is the model that estimates it.

Question: How well does Elo rating estimate a player's strength?

Bonus question: What happens if you take the player's rating curve into account?

• It shouldn't be that hard to measure this (take a large database of games, filter out those that have missing ratings or are only a single move, put them into bins depending on rating difference, and then check if the average score inside each bin is close to what Elo predicts) but I can't find any good link right now. My gut feeling is that the estimation is very good indeed. – RemcoGerlich Mar 3 '14 at 10:12
• While I've always found ELO to be a fair representation of rating (and a great statistical tool), at an individual level there are many other considerations that it (obviously) doesn't take into account. I've experienced cases where 3 people of similar overall rating had different styles, such that they could generally beat each other in a circular fashion: A < B, B < C and C < A. I'd say that this kind of thing is the exception though, not the rule. – Daniel B Mar 6 '14 at 13:05

The Elo rating is a statistical prediction of your results against players with known rating when they play each other in the long term. Some examples:

• Two players with the same rating will score the same amount of victories when playing together.

• Two players with a 200 Elo difference, means the stronger player will win 3 out of 4 games (and lose the other).

• Two players with a 400 Elo difference, means the stronger player will win 15 out of 16 times (and the weakest winning the other one).

• And so on, a difference beyond 700 is not accurately measurable (basically the system predicts 100% score of the stronger player).

Now, in practice, many other factors come into play (ie. the stronger player being bored or overconfident, playing a system in which the weakest players has in-depth knowledge, being in shape, etc.)

For more details, see

http://www.thechessworld.com/learn-chess/18-general-information/136-deeper-look-at-elo-chess-rating-system-

http://en.wikipedia.org/wiki/Elo_rating

It's not clear to me whether you are asking a philosophical or a quantitative question.

My answer to the philosophical question is that the Elo rating is predictive; that is, it is designed to make accurate predictions about the outcomes of chess games. To that extent, it measures strength by estimating the likelihood that a player will win against another player. It does this by using the player's past results, and it does the best it can given that information.

The answer to the quantitative question is that the "expected value" of a game between two players is correlated with the difference in their Elo ratings. For example if player A's Elo rating is 200 higher than that of player B, then player A would be expected to go about 76-24 over the course of a 100-game match.

Note: The question has changed since I answered it, so my answer makes less sense now. The title of the question used to be Does the chess rating represent a player's strength?, and the question was "I notice that somehow, we chess players associate the chess rating with strength. But is it a measure of strength? To what extent?" Please take this answer as a response to that question, not the one it has been changed to.

• I think of strength as the "real" that is predicted by a "model" (in this case, the rating). – Rauan Sagit Mar 2 '14 at 0:47
• The question is, how good is its estimation? There are other systems (like Glicko) that claim to be better. – RemcoGerlich Mar 3 '14 at 10:13
• @RemcoGerlich That wasn't the question at the time :) See my edit. – dfan Mar 3 '14 at 13:20
• Yeah, sorry, at first I didn't notice that this was an edited question. – RemcoGerlich Mar 3 '14 at 13:24
• @dfan Yep. I edited the question because I thought it would be more interesting this way. After all, FIDE Elo rating is the official rating of chess players. It is custom to make assumptions about a player's strength based on their Elo rating. Yet, how accurate is this estimation method? – Rauan Sagit Mar 5 '14 at 15:07

"Normally" (pun intended) the Elo rating does indicate a player's average strength (you tagged this post with Elo, so I am considering Elo rating).

Elo rating assumes that a player's performance conforms to what is called the Normal Distribution. Thus, if your rating is 2700, then that's your mean or average level of performance. However, that does not give an indication of how far varied your performance can be in relation to your mean rating.

For instance, if your standard deviation is low, then it means that your performance is very consistent around 2700 (for example, you beat most guys below 2600, hardly ever beat 2800+, and usually draw with players between that range). If your standard deviation is high, then it means that sometimes, you beat even Magnus Carlsen (2881 as of today), but at times you find yourself losing to a low rated Grandmaster or even an International Master.

• It does no such thing. Rather it is much closer to a primitive form of logistic regression, except the computations can be carried out by hand. – shabbychef Jan 6 '19 at 1:41
• Not quite. The normal distribution goes from negative infinity to plus infinity. The proper distribution would be a beta type with finite values for upper and lower actual ratings. – yobamamama Jan 9 '20 at 18:33

The Elo rating system doesn't estimate a player's strength as well as other systems. I've read a while ago about a contest between rating systems, I'll describe it by quoting the article: competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games.

The conclusion:

The Elo benchmark was outperformed within 24 hours

On a sidenote, I can't see the charts anymore, if anyone finds them, you're welcome to contribute them. From what I remember there were no less than 60 rating systems which outperformed Elo.

• 8631 top players will in no way represent what the vast bulk of more ordinary players would actually do. – yobamamama Jan 9 '20 at 18:35

The weakness of the Elo rating system is that it depends on scoring, that is the assignment of win=1, loss=0, and draw=1/2. Scoring causes some information about chess game outcomes to be lost. It would be more accurate to measure chess playing strength in two dimensions. Please see: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2477868

• The real weakness is that there are not enough games among a much wider range of opponents to swamp out the locality differences. Some cities have much stronger players than others yet their ratings tend to be the same as they play each other in their city and rarely play others from the other city. – yobamamama Jan 9 '20 at 18:38

An Elo rating most emphatically is NOT a measure of a player's strength, nor was it intended to be. A player might, for all kinds of reasons, have a lower Elo rating than someone that is in reality weaker. For example a player that plays mostly in team competitions might accept a draw in a winning position, or risk losing by playing for a win in a drawn position, with bad effects on their rating.

Another example: One player in SWISS tournaments might play objectively to get the "correct" number of points that their play deserves. Another might take great risks on the grounds that 3.5 or 4 out of 6 finishes nowhere whereas 5 or more is usually in the money, and worth the risk of losing from an easily drawn position. The first player would probably earn a higher rating, but the second would win more tournaments! (and might actually be stronger).

The maths behind Elo ratings guarantees that rating points are gained more slowly than is warranted by current form and also ensures that a few bad results do not reduce a rating by a huge amount. In effect games played years earlier when a player was much weaker (or stronger) affect the rate at which the rating can change. This aspect of controlling the rate of change is further fudged by choice of k-factor.

A further problem is that Elo ratings only apply properly to players that are contemporary. They do not give meaningful comparisons of strength for players active in different eras. Of course that has not stopped people from making such comparisons and concluding (for example) that Lasker and ZUuckertort would hold their own against today's GM's that former world-champion Petrosian was no stronger than any modern-day top-50 super-GM, or that Fischer was significantly weaker than Kasparov and Carlsen. Those conclusions may or may not be true, but it is just wrong to use Elo ratings (retrospectively calculated as necessary) to arrive at them.

What Elo ratings are designed for is to allow the probability of encounters between players to be estimated. They do that quite well for players that have stable ratings that have not changed much over a few years. They do a poor job when one or both of the players is either improving or declining.

They have been adopted as a proxy for chess strength, but are far from perfect. The right way to assess strength is to study a players games and make a judgement, but to do that you have to be very strong yourself, and it is hard work.

As in so many other areas of life and human endeavour it is much easier to pretend that a simple number based on results tells us what we'd like to know ... even when it doesn't.

One interpretation of the question is "What is the standard error of Elo?" That is, if Elo is a statistic that measures some true latent ability, what is the standard deviation of that statistic? If, for example, you've just played 50 tournament games, and your Elo is calculated as 1700, is it really 1700, or might it actually be 1710? Or 1750? Might it be 1600?

I have analyzed this question in a blog post, the gist of which is as follows: Suppose two players play each other repeatedly. The difference in true abilities is "Delta A", where we assume the probability of a win by the first player is the Elo logistic function in Delta A. Then, after many games between them, the difference in their computed Elos will be Delta r which equals Delta a plus some error, epsilon. That error term, epsilon, approximately follows an autoregressive process, and is asymptotically unbiased. The standard error depends on the "k factor" used in updating Elo, and on the probability of a tie (which is typically higher for better players, when equally matched). Here is a plot of the empirical (dots) and theoretical standard error versus the k factor, for different probabilities of a tie: The upshot is that differences in Elo will have standard error around 20 or more.

As others have noted, the Elo rating has some deficiencies. However, it is fairly simple to describe and compute, which was necessary at the time of its introduction.