The FIDE rating of very active chess players can vary a lot: rating fluctuations of 50 rating points are not unusual.
Is there a way to distinguish whether a rating rise/drop can be explained by random factors, e.g. "(bad) luck", being "in/out form", etc..., or an actual increase/decrease in playing strength?
Does it make sense to say "a rating increase of X over minimum Y games is statistically significant"? If yes, what is the value of X and Y?
Is there a way to distinguish whether a rating rise/drop can be explained by random factors, e.g. "(bad) luck", being "in/out form", etc..., or an actual increase/decrease in playing strength?
No, would be my answer, at least there is no easy way to distinguish.
The problem is that there are very many factors affecting ratings.
For instance, about 5 years ago FIDE changed the system. I'm not sure if it was lowering the minimum rating, changing the k factors or a combination of these but the net effect was a rating deflation. Talking to a range of players it seems that there was an influx of very underrated juniors into the system. One disgruntled player complained of losing or drawing to juniors who were, in his opinion, as much as 400 points underrated.
Over the following 2 years I noticed a 150 point drop in my rating and anecdotal evidence is that many other older players who have had stable ratings for many years noticed the same kind of drop. Over the last year these same players have reported that their ratings are rising back towards their old averages and attribute this to the system regarding rating of junior players (perhaps with their higher k value) as "settling down". I have to admit I haven't noticed this effect yet ;-), but then I played very few games in the last year. I hope this means I have something to look forward to if I go back to playing a lot more games!
You can also have anomalies due to the average strength of the players you play against. If you play a lot of players who are much, much weaker or much, much stronger than you then this can also distort your rating due to the way they are calculated.
Finally, I would just add that FIDE only attempts to measure my rating "sitting down". My rating "standing up" is at least 300 points higher ;-).
This can be a very difficult problem in practice, depending on your expectations. You would firstly need to assume that in no games were there blunders, and already you can see problems begin to crop up as it is difficult to define what constitutes a blunder (Win to loss? Win to draw? Queen blunder? Positional blunder? Touch-move?).
If you make the further assumption that all of a player's opponents are 'fairly' rated (although this is something fairly dubious to say), then it may be somewhat possible to come up with statistical tests. But first off, it is better to rephrase the question from "X rating points in Y games" to "X game points from Y games", because rating changes depend on the K-value.
Consider the following: you have n games, and are expected to win "rating change"/"K-value" of them. You can also compute the expected score for each game (which depends on rating difference), and use these as your probabilities in a sequence of multinomial trials. For example, say you have 5 opponents, with rating difference relative to you and "win expected" as follows:
+43, WE=0.44
-12, WE=0.52
+135, WE=0.32
+22, WE=0.47
-50, WE=0.57
Then your total wins expected is 2.32. Constructing a statistical test here is tricky, because you can't use the binomial distribution properly (as it assumes that only two results are possible, whereas in practice, a third result, a draw, is also possible), and the Elo rating system only gives you the expected score, and not the win/draw/loss probabilities, which depends on the absolute value of the players' rating (i.e. a draw is much more likely between 2700 vs. 2800 than 1700 vs. 1800) as well as colours (whose significance, again, depends on absolute rating)!
You could, of course, go for a naive test and ignore the final factors, but if your total score contains a half point, you won't even be able to use this test!
Anyway, supposing you have the easy case of a sequence of wins/losses without draws, say you scored 3/5 in the above example. Then for your p-value, you would need to compute the probability of scoring 3 points or more, which you get by summing over the probabilities of all possible ways to score 3 points or more (wins vs. players 123, 124, ..., 345, 1234,..., 2345, 12345) which would take quite some time to be done by hand (you need to calculate and sum 5C3+5C4+5C5=16 products of 5 numbers), and if this sum is less than some critical value, say 0.05, then according to this very naive test, your actual performance is indeed better than your rating suggests.
Actually, the test would not be too bad if you knew the drawing probabilities: you can do the above again, though there will be much more work; the assistance of a computer is definitely necessary.
At the end of the day, the best test is time and exposure. Play in international tournaments, and if you maintain your rating change over time, then your playing strength has certainly changed.
(Technically, the tests I described are one-tailed, and you might need to consider two-tailed tests, but deciding on which to use and their impact is relatively trivial and unimportant compared with the other problems you need to address!)
The man who invented the rating system (Arpad Elo) likened it to measuring the depth of a stream using a yardstick that was hanging from a tree branch being blown in the wind. Elo himself didn't think differences of less than 100 points were even worth trying to measure. The only reason the system was worked out as a four-digit number in the first place was because the USCF rating system he was trying to fix was a four-digit system; he would have been just as happy with two-digit numbers.
Too many things affect the outcome of a chess game, including slight variations in health and the current physical condition of both players (one may be tired, didn't sleep well, etc.) for anyone to be able to characterize the strength of any chess player with any sort of precision. A rise in rating may mean your opponents have been playing below par, or it may mean they were playing above par and your skill actually improved far better than the rise indicates. The pertinent question is, does chess make more sense to you now than it did? Ae you seeing more, and understanding more? Ratings can't answer those questions. Only you can.
If such a method exists, it is either very difficult, very subjective or very unreliable. If it was easy, objective and reliable, FIDE would incorporate it in their rating system.
More pointedly, your question makes the assumption that a player has a "true objective actual" strength and that his Elo is a (possibly fallible) approximation of that strength. This assumption is debatable in itself: does every player have a true "real" rating? Or is it more nuanced than that? I am of the opinion that our playing ability on any given day is influenced by a large number of psychological and environmental factors, which means that you can never say "my REAL rating is x even if my official rating is y". As philosophers sometimes like to say, there is no "fact of the matter" of what your actual rating is.
Update: to everybody who thinks this is just a matter of statistics, here is a challenge. Last year at the Sinquefield Cup, Caruana won 7 games in a row and increased his "live" rating to over 2850. Show how statistics can be used to prove that it was just a random streak, or whether he actually had an improved playing strength over that period.
The problem with using elo as a statistical measure is that it forces you to make assumptions that are difficult to justify. Can you model a player's strength as a constant probability distribution? Or do players' strength change over time? Is this change frequent, stuttery or smooth? Do elo's expected results match up with history (empirical studies shows that elo becomes less reliable when rating difference is large). There are many others: https://en.m.wikipedia.org/wiki/Elo_rating_system#Mathematical_issues
Elo is a useful agreed upon number that is easy to calculate, and we think it approximates playing strength well enough to organize our tournaments around it. But be careful when making grand claims about what it can tell us about the state of our mental abilities without adding a big disclaimer.
When your new rating, over a large number of games , has a minimum rating that is higher than your previous highest rating then you are sure you have improved. A handful of games and certainly less than a year in elapsed time unless you play tournaments every month if not more often are not that significant.
