6

In season 14 of TCEC, Stockfish defeated Leela 50.5-49.5 in a 100-game match (the closest possible winning margin). Using an elo calculator such as this one, we can see that the elo difference between the two engines was +3.

However it must be that this isn't a statistically significant result - after all the match was only 100 games, and it's the smallest possible winning margin. Therefore the error bars on this calculation should be at least 3. The calculator doesn't give the error bars though, and I can't find any equations on how to calculate the error. How do you calculate the error on this estimate?

Related: How do you calculate elo?

Edit: Looks like 3dkingdoms have improved their calculator, which is now capable of calculating an error margin. I'd still be interested in seeing how that error margin is calculated, though.

Improve this question
8
  • 6
    I'm voting to close this question as off-topic because it is a maths question, not a chess question.
    – Brian Towers ♦
    Commented May 23, 2019 at 9:09
  • 1
    LOL. People ask questions about shogi and things like that all the time
    – David
    Commented May 23, 2019 at 20:38
  • 9
    I'd put this on Math.SE, but then first I'd have to explain what elo is, how it translates to expected performance, etc.
    – Allure
    Commented May 23, 2019 at 21:29
  • 2
    @Allure I think CrossValidated (Stats.SE) can handle this if Chess.SE won't allow it. You could probably reword it to "What is the uncertainty on the rating estimate in the Elo model? in order to make it suitable here.
    – SecretAgentMan
    Commented Feb 13, 2023 at 13:08
  • 5
    Voting to reopen. If the math involved here isn't sufficiently chess related, then neither is almost any other "chess question".
    – Charles Rockafellor
    Commented Feb 13, 2023 at 16:20

2 Answers 2

Reset to default
1

Elo calculations are not a statistical estimate of an unknown parameter, they are just an aggregate count of actual results. Since all those results are known, the "error bar" would have size 0.

It is possible to perform a test of statistical significance over match results to determine if there's enough evidence to conclude that one engine/player is stronger than the other, but that's a separate problem from Elo computation.

Improve this answer
4
  • It is possible to perform a test of statistical significance over the match results to determine if there's enough evidence to conclude that one engine/player is stronger than the other Do you know how to do this test, e.g. with the match in the OP as an example?
    – Allure
    Commented Apr 14, 2023 at 9:07
  • 1
    @Allure with the information we have the best choice would probably be a proportions test (statology.org/one-proportion-z-test). Let p be the average score Stockfish would get against Leela and let's take p=0.5 as the null hypothesis. However since chess is a game with three possible results, it'd be more accurate to have the win/draw/loss count rather than just the final score. Then we can check if the number of wins is significantly bigger than the number of losses instead of just a test over the final score.
    – David
    Commented Apr 14, 2023 at 9:12
  • @David To get to the spirit of the question, would it be valid to just go to statology.org/one-proportion-z-test-calculator and fill in p=0.505 and n=100, look at the 95% confidence intervals, and plug them into the FIDE fractional-score-to-rating table to convert them to a ratings difference? Or does the 3 result possibility throw this calculation off? (If I do that, I get 95% C.I. = [0.4070, 0.6030] which gives me about -67 to +74.)
    – D M
    Commented Apr 15, 2023 at 1:11
  • @DM In this case it'd be the same since even if it was 1 win and 99 draws you wouldn't get a significant difference. The proportions test approximation is good enough. One could also try a t-test for normal distributions as a reasonable approximation
    – David
    Commented Apr 15, 2023 at 11:08
0

Here is a practical answer: Since, as already said, the calculation is "exact", look instead at the rating fluctuations of players since by any model, the rating of a (medium aged) player is constant, modulo local form fluctuations, aging and whatnot. These random fluctuations should be a far better proxy for the "real" sigma of the Elo model than any contrieved math. (Only don't take my charts :-)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.