Rating rise: luck or actual increase in playing strength?

The FIDE rating of very active chess players can vary a lot: rating fluctuations of 50 rating points are not unusual.

Is there a way to distinguish whether a rating rise/drop can be explained by random factors, e.g. "(bad) luck", being "in/out form", etc..., or an actual increase/decrease in playing strength?

Does it make sense to say "a rating increase of X over minimum Y games is statistically significant"? If yes, what is the value of X and Y?

• It's very easy: Every increase is actual strength, every decrease is bad luck or being out of form. Commented Aug 8, 2015 at 9:22
• More seriously: X and Y will differ between players. They depend on the variance of a players results. Commented Aug 8, 2015 at 9:25
• I agree, I wonder how to apply it on an example? Say, a player's rating fluctuates between 2050 and 2100. Suddenly, his rating quickly rises to 2150 (indicating an increase in playing strength). Then, he quickly drops back to 2100 (now, the rise to 2150 should be seen as a fluctuation). So, one can only say at hindsight whether a rating increase was "luck" or not? Commented Aug 8, 2015 at 10:22
• @Maxwell - how do you know that when his rating went to 2150 he was not actually then playing to his "true potential"? Maybe his wife went on holiday, allowing him more time to prep, but now she's back and the responsibilities of his life is making him mediocre again. Commented Aug 8, 2015 at 17:27

Is there a way to distinguish whether a rating rise/drop can be explained by random factors, e.g. "(bad) luck", being "in/out form", etc..., or an actual increase/decrease in playing strength?

No, would be my answer, at least there is no easy way to distinguish.

The problem is that there are very many factors affecting ratings.

For instance, about 5 years ago FIDE changed the system. I'm not sure if it was lowering the minimum rating, changing the k factors or a combination of these but the net effect was a rating deflation. Talking to a range of players it seems that there was an influx of very underrated juniors into the system. One disgruntled player complained of losing or drawing to juniors who were, in his opinion, as much as 400 points underrated.

Over the following 2 years I noticed a 150 point drop in my rating and anecdotal evidence is that many other older players who have had stable ratings for many years noticed the same kind of drop. Over the last year these same players have reported that their ratings are rising back towards their old averages and attribute this to the system regarding rating of junior players (perhaps with their higher k value) as "settling down". I have to admit I haven't noticed this effect yet ;-), but then I played very few games in the last year. I hope this means I have something to look forward to if I go back to playing a lot more games!

You can also have anomalies due to the average strength of the players you play against. If you play a lot of players who are much, much weaker or much, much stronger than you then this can also distort your rating due to the way they are calculated.

Finally, I would just add that FIDE only attempts to measure my rating "sitting down". My rating "standing up" is at least 300 points higher ;-).

• Thanks for pointing out a couple of reasons that makes it very difficult in practice. And good luck with your future games and winning those rating points back! Commented Aug 8, 2015 at 18:26
• @Maxwell86 Thanks! Another example of rating vagaries happened to me in a rapid tournament I played recently. I scored 3/6. All 3 of my losses were to higher rated opponents who already had rapid ratings. All 3 of my wins were against players who didn't have rapid ratings even though one of them had a higher standard rating than me. Result from a reasonable 3/6 performance was that my rapid rating took a hammering :-( Commented Aug 8, 2015 at 18:45

This can be a very difficult problem in practice, depending on your expectations. You would firstly need to assume that in no games were there blunders, and already you can see problems begin to crop up as it is difficult to define what constitutes a blunder (Win to loss? Win to draw? Queen blunder? Positional blunder? Touch-move?).

If you make the further assumption that all of a player's opponents are 'fairly' rated (although this is something fairly dubious to say), then it may be somewhat possible to come up with statistical tests. But first off, it is better to rephrase the question from "X rating points in Y games" to "X game points from Y games", because rating changes depend on the K-value.

Consider the following: you have n games, and are expected to win "rating change"/"K-value" of them. You can also compute the expected score for each game (which depends on rating difference), and use these as your probabilities in a sequence of multinomial trials. For example, say you have 5 opponents, with rating difference relative to you and "win expected" as follows:

+43, WE=0.44

-12, WE=0.52

+135, WE=0.32

+22, WE=0.47

-50, WE=0.57

Then your total wins expected is 2.32. Constructing a statistical test here is tricky, because you can't use the binomial distribution properly (as it assumes that only two results are possible, whereas in practice, a third result, a draw, is also possible), and the Elo rating system only gives you the expected score, and not the win/draw/loss probabilities, which depends on the absolute value of the players' rating (i.e. a draw is much more likely between 2700 vs. 2800 than 1700 vs. 1800) as well as colours (whose significance, again, depends on absolute rating)!

You could, of course, go for a naive test and ignore the final factors, but if your total score contains a half point, you won't even be able to use this test!

Anyway, supposing you have the easy case of a sequence of wins/losses without draws, say you scored 3/5 in the above example. Then for your p-value, you would need to compute the probability of scoring 3 points or more, which you get by summing over the probabilities of all possible ways to score 3 points or more (wins vs. players 123, 124, ..., 345, 1234,..., 2345, 12345) which would take quite some time to be done by hand (you need to calculate and sum 5C3+5C4+5C5=16 products of 5 numbers), and if this sum is less than some critical value, say 0.05, then according to this very naive test, your actual performance is indeed better than your rating suggests.

Actually, the test would not be too bad if you knew the drawing probabilities: you can do the above again, though there will be much more work; the assistance of a computer is definitely necessary.

At the end of the day, the best test is time and exposure. Play in international tournaments, and if you maintain your rating change over time, then your playing strength has certainly changed.

(Technically, the tests I described are one-tailed, and you might need to consider two-tailed tests, but deciding on which to use and their impact is relatively trivial and unimportant compared with the other problems you need to address!)

• Yes, "the best test is time and exposure", only then you'll know, that's probably the conlusion. And if there are no anomalies like the ones Brian Towers mentioned... Commented Aug 8, 2015 at 19:05

The man who invented the rating system (Arpad Elo) likened it to measuring the depth of a stream using a yardstick that was hanging from a tree branch being blown in the wind. Elo himself didn't think differences of less than 100 points were even worth trying to measure. The only reason the system was worked out as a four-digit number in the first place was because the USCF rating system he was trying to fix was a four-digit system; he would have been just as happy with two-digit numbers.

Too many things affect the outcome of a chess game, including slight variations in health and the current physical condition of both players (one may be tired, didn't sleep well, etc.) for anyone to be able to characterize the strength of any chess player with any sort of precision. A rise in rating may mean your opponents have been playing below par, or it may mean they were playing above par and your skill actually improved far better than the rise indicates. The pertinent question is, does chess make more sense to you now than it did? Ae you seeing more, and understanding more? Ratings can't answer those questions. Only you can.

If such a method exists, it is either very difficult, very subjective or very unreliable. If it was easy, objective and reliable, FIDE would incorporate it in their rating system.

More pointedly, your question makes the assumption that a player has a "true objective actual" strength and that his Elo is a (possibly fallible) approximation of that strength. This assumption is debatable in itself: does every player have a true "real" rating? Or is it more nuanced than that? I am of the opinion that our playing ability on any given day is influenced by a large number of psychological and environmental factors, which means that you can never say "my REAL rating is x even if my official rating is y". As philosophers sometimes like to say, there is no "fact of the matter" of what your actual rating is.

Update: to everybody who thinks this is just a matter of statistics, here is a challenge. Last year at the Sinquefield Cup, Caruana won 7 games in a row and increased his "live" rating to over 2850. Show how statistics can be used to prove that it was just a random streak, or whether he actually had an improved playing strength over that period.

The problem with using elo as a statistical measure is that it forces you to make assumptions that are difficult to justify. Can you model a player's strength as a constant probability distribution? Or do players' strength change over time? Is this change frequent, stuttery or smooth? Do elo's expected results match up with history (empirical studies shows that elo becomes less reliable when rating difference is large). There are many others: https://en.m.wikipedia.org/wiki/Elo_rating_system#Mathematical_issues

Elo is a useful agreed upon number that is easy to calculate, and we think it approximates playing strength well enough to organize our tournaments around it. But be careful when making grand claims about what it can tell us about the state of our mental abilities without adding a big disclaimer.

• I would say this is just statistics. You have a bunch of measurements and say something about future measurements. Numbers, not philosophy. And this is something a statistician shouldn't have much problems with. Commented Aug 8, 2015 at 9:55
• Your rating is the statistical measurement. The question is, how do we know if the statistics reflect "real" strength? My answer says we don't - the statistics is all we have. I would be very interested to see a statistical measurement of "bad luck" vs "bad form". Commented Aug 8, 2015 at 10:28
• Rating is assumed to be unbiased. Random factors can be measured statistically. For example, we can measure the number of standard deviation away. The change can be tested with a null hypothesis. Alternatively, we can do a bayesian analysis where we have some kind of prior distribution. Commented Aug 8, 2015 at 10:35
• You can come up with all the statistics you want, the question of whether your statistics measure a player's "true real" strength will still not be answered until you account for the myriad of factors that determines a player's performance over any given period. "Was I unlucky, was my form bad, did my strength weaken or did my opponents drink gummie berry juice"? Statistics can only tell you what we already know: my performance (as measured by this choice of measurement) this month was different from last month. Commented Aug 8, 2015 at 10:59
• Well, I'm not a statistician, but the fact alone that since St Louis Grischuk, Carlsen, Giri and Kramnik had had six game streaks and Caruana had a five game streak shows that Caruana didn't have to turn into a 3000 Elo player to pull this off. If you add the statistic that Caruana has the lowest draw percentage of the elite, it becomes even less of a surprise. As I said, I don't know much about statistics, but I don't see why this shouldn't be described mathematically. Commented Aug 9, 2015 at 14:31

When your new rating, over a large number of games , has a minimum rating that is higher than your previous highest rating then you are sure you have improved. A handful of games and certainly less than a year in elapsed time unless you play tournaments every month if not more often are not that significant.