I originally had this posted on Stack Overflow as it could be a bug in the implementations, but some suggested I post to math. I just found this Stack Exchange, and I thought who better? Some of you may know off the bat whether this seems accurate or not, without the need for debugging.
Glicko-2 is a rating system used in chess but can be used in many other situations. Glicko-2 is an improvement on Glicko-1, which addressed problems of the older ELO rating.
What makes Glicko-2 special in comparison to version 1 is that it incorporates a higher rating deviation (RD) the longer someone has been inactive. It does this with the notion of a system constant which relates to time/rating periods.
An example write-up from the author is found here: http://www.glicko.net/glicko/glicko2.pdf.
Within this document, he explains:
The Glicko-2 system works best when the number of games in a rating period is moderate to large, say an average of at least 10-15 games per player in a rating period. The length of time for a rating period is at the discretion of the administrator.
Making an assumption that a group of active chess players plays 10-15 games on average in a 1 month time period, the administrator would then update ratings at the end of every month.
I needed a PHP Implementation of the Glicko-2 rating system and came across the following:
Glicko-2 JavaScript Implementation
- The JavaScript had a small error, which didn't let it match the technical write-up example, the author found it close enough, and didn't bother to debug.
- The PHP implementation was plagued with many bugs, but that wasn't apparent unless you did more than one rating period (which the technical write-up never shows expected values of)
- Finally, the Excel calculator seemed to be error-free and the most professional, done by someone in the chess community. Once the JavaScript bug was solved, the JavaScript and Excel Calculator matched very closely with each other (albeit not perfect, could be within rounding error)
I had fixed the bugs (and submitted issues/patches to the authors) I could find on the PHP and JavaScript versions to match as closely to the Excel Calculator.
Now I am 99% confident that I have an accurate Glicko-2 implementation (between the 3 of them) for analysis and that is when I came across something strange, and the topic of this discussion.
Given the suggested default for Glicko-2 for a new player:
Rating: 1500
RD: 350
Volatility: 0.06
If you face an average opponent of rating 1378 and RD 99 (source) only once every rating period (1 month) for the next 12 periods (1 year) you will have accumulated an assumed National Class A (1800-1999) rating of 1852 when in reality you have only beaten 12 average rated players over a span of 12 months.
Month Rating RD Volatility Class
1 1625 259 0.059999 National Class B
2 1682 225 0.059998 〃
3 1718 205 0.059997 〃
6 1784 174 0.059994 〃
12 1852 148 0.059988 National Class A
24 1922 127 0.059976 〃
If you face 2 average opponents every rating period, you can get to National Class A about 4-5 months, facing only 8-10 average opponents.
Month Rating RD Volatility Class
1 1672 215 0.059999 National Class B
2 1733 183 0.059997 〃
3 1770 166 0.059995 〃
4 1797 154 0.059993 〃
5 1819 146 0.059992 National Class A
6 1836 140 0.059991 〃
Are these assumptions accurate? Is there a bug in my calculator?
If it is not a bug, what are some ways of countering this besides:
- Consider "true rating" to be lower bound of the deviation (Rating - RD)
- Do not show inactive user's rating
- Do not show users with less than N games