Glicko-2 Rating System: Bug or exploit?

Question

I originally had this posted on Stack Overflow as it could be a bug in the implementations, but some suggested I post to math. I just found this Stack Exchange, and I thought who better? Some of you may know off the bat whether this seems accurate or not, without the need for debugging.

---

Glicko-2 is a rating system used in chess but can be used in many other situations. Glicko-2 is an improvement on Glicko-1, which addressed problems of the older ELO rating.

What makes Glicko-2 special in comparison to version 1 is that it incorporates a higher rating deviation (RD) the longer someone has been inactive. It does this with the notion of a system constant which relates to time/rating periods.

An example write-up from the author is found here: http://www.glicko.net/glicko/glicko2.pdf.
Within this document, he explains:

The Glicko-2 system works best when the number of games in a rating period is moderate to large, say an average of at least 10-15 games per player in a rating period. The length of time for a rating period is at the discretion of the administrator.

Making an assumption that a group of active chess players plays 10-15 games on average in a 1 month time period, the administrator would then update ratings at the end of every month.

---

I needed a PHP Implementation of the Glicko-2 rating system and came across the following:

Glicko-2 JavaScript Implementation

• The JavaScript had a small error, which didn't let it match the technical write-up example, the author found it close enough, and didn't bother to debug.

Glicko-2 PHP Implementation

• The PHP implementation was plagued with many bugs, but that wasn't apparent unless you did more than one rating period (which the technical write-up never shows expected values of)

Glicko-2 Calculator in Excel

• Finally, the Excel calculator seemed to be error-free and the most professional, done by someone in the chess community. Once the JavaScript bug was solved, the JavaScript and Excel Calculator matched very closely with each other (albeit not perfect, could be within rounding error)

I had fixed the bugs (and submitted issues/patches to the authors) I could find on the PHP and JavaScript versions to match as closely to the Excel Calculator.

---

Now I am 99% confident that I have an accurate Glicko-2 implementation (between the 3 of them) for analysis and that is when I came across something strange, and the topic of this discussion.

Given the suggested default for Glicko-2 for a new player:

Rating:      1500
RD:           350
Volatility:  0.06

If you face an average opponent of rating 1378 and RD 99 (source) only once every rating period (1 month) for the next 12 periods (1 year) you will have accumulated an assumed National Class A (1800-1999) rating of 1852 when in reality you have only beaten 12 average rated players over a span of 12 months.

Month   Rating      RD      Volatility      Class
1       1625        259     0.059999        National Class B
2       1682        225     0.059998        〃
3       1718        205     0.059997        〃
6       1784        174     0.059994        〃
12      1852        148     0.059988        National Class A
24      1922        127     0.059976        〃

If you face 2 average opponents every rating period, you can get to National Class A about 4-5 months, facing only 8-10 average opponents.

Month   Rating      RD      Volatility      Class
1       1672        215     0.059999        National Class B
2       1733        183     0.059997        〃
3       1770        166     0.059995        〃
4       1797        154     0.059993        〃
5       1819        146     0.059992        National Class A
6       1836        140     0.059991        〃

---

Are these assumptions accurate? Is there a bug in my calculator?

If it is not a bug, what are some ways of countering this besides:

• Consider "true rating" to be lower bound of the deviation (Rating - RD)
• Do not show inactive user's rating
• Do not show users with less than N games

Answer 1

I worked on a Scala implementation a few months ago, although it was a bit unpolished--I should go back to finish it. I at least got some reasonable results from it.

If you win every game you play, yes, your rating will grow to be quite high even if you play against only low rated players. The probability that you will win every game against such an opponent is probably about what a class A player would achieve (although it probably is a bit inflated due to the relatively high RD, still).

The best way to counter, in my opinion, is to not consider someone with an RD over a certain amount to be a stable rating--i.e. consider it to be "provisional". Also, in order to actually gain a title, at least in USCF, they have a system of norms, where you must perform at a certain level in a tournament of 4 games or more (4 times, I believe), which makes the likelihood of playing against a ~1378 for the whole tournament [four times] very unlikely.

Is your goal to use this for chess? What is your use case?

Update: FICS handles it by only considering people with an RD of <80 active. (They use Glicko-1, still, I believe.) http://www.freechess.org/Help/ficsfaq.html#Q005.003

And by the way, Glicko-1 uses RD/time decay as well. Glicko-2's main improvement was the "volatility" factor, which allows people with erratic results or stable results to be calculated very slightly differently. I think it's a very minor tweak to Glicko-1 which causes considerable extra calculation--but like you, I was still interested in calculating it. I actually asked Glickman himself for some additional datapoints for testing, but he was too busy to supply them at the time.

Answer 2

All rating systems have problems in real life. At best none of them can do more than approximate a guess at your ability.

They will do better at the GM level as those players are more consistent and tend to play more GMs; while the rest of us are more inconsistent and tend to play a wider range of lower rated players who are also more inconsistent. And new players tend to improve far faster while playing in fewer tournaments which further skews all the ratings. And at any level some players have off days or do not try as hard if it wont change their position in the final results.

Trying to fine tune ratings so much like Glicko2 is a fools errand.

Another factor exacerbating the problem is that players tend to play the same players. If there were a worldwide mandatory swiss system that played enough rounds there would be a big shake up in most players ratings. A worldwide double round robin would make some more changes but the swiss would be a good start to fixing rating errors -- at least in the middle with established players. You still have the problem with new players distorting the ratings.