Glicko-2 Rating System: Bug or exploit? I originally had this posted on StackOverflow as it could be a bug in the implementations, but some suggested I post to math. I just found this stackexchange, and I thought who better? Some of you may know off the bat rather this seems accurate or not, without the need of debugging. Any links to online calculators or alternative rating methods are welcomed.



Glicko-2 is a rating system used in chess, but can be used in many other situations. Glicko-2 is an improvement on Glicko-1, which addressed problems of the older ELO rating.

What makes Glicko-2 special in comparison to version 1  is that it incorporates a higher rating deviation (RD) the longer someone has been inactive. It does this with the notion of a system constant which relates to time/rating periods.

An example write up from the author is found here: http://www.glicko.net/glicko/glicko2.pdf.
Within this document, he explains:


  The Glicko-2 system works best when the number of games in a rating
  period is moderate to large, say an average of at least 10-15 games
  per player in a rating period. The length of time for a rating period
  is at the discretion of the administrator.


Making an assumption that a group of active chess players play 10-15 games on average in a 1 month time period, the administrator would then update ratings at the end of every month.



I needed a PHP Implementation of the Glicko-2 rating system and came across the following:

Glicko-2 JavaScript Implementation


The JavaScript had a small error, in which didn't let it match the
technical write-up example, the author found it close enough, and
didn't bother to debug.


Glicko-2 PHP Implementation


The PHP implementation was plagued with many bugs, but that wasn't
apparent unless you did more than one rating period (which the
technical write-up never shows expected values of)


Glicko-2 Calculator in Excel


Finally the Excel calculator seemed to be error-free and the most
professional, done by someone in the chess community. Once the
JavaScript bug was solved, the JavaScript and Excel Calculator
matched very closely with each other (albeit not perfect, could be within rounding error)


I had fixed the bugs (and submitted issues/patches to the authors) I could find on the PHP and JavaScript versions to match as closely to the Excel Calculator



Now I am 99% confident that I have an accurate Glicko-2 implementation (between the 3 of them)  for analysis and that is when I came across something strange, and the topic of this discussion.

Given the suggested default for Glicko-2 for a new player:

Rating:      1500
RD:           350
Volatility:  0.06


If you face an average opponent of rating 1378 and RD 99 (Source) only once every rating period (1 month) for the next 12 periods (1 year) you will have accumulated an assumed National Class A (1800-1999) rating of 1852 when in reality you have only beat 12 average rated players over a span of 12 months. 

Month   Rating      RD      Volatility      Class
1       1625        259     0.059999        National Class B
2       1682        225     0.059998        〃
3       1718        205     0.059997        〃
6       1784        174     0.059994        〃
12      1852        148     0.059988        National Class A
24      1922        127     0.059976        〃


If you face 2 average opponents every rating period, you can get to National Class A about 4-5 months, facing only 8-10 average opponents.

Month   Rating      RD      Volatility      Class
1       1672        215     0.059999        National Class B
2       1733        183     0.059997        〃
3       1770        166     0.059995        〃
4       1797        154     0.059993        〃
5       1819        146     0.059992        National Class A
6       1836        140     0.059991        〃



Are these assumptions accurate? Is there a bug in my calculator? 

If it is not a bug, what are some ways of countering this besides: 


Consider "true rating" to be lower bound of the deviation (Rating -
RD)
Do not show inactive user's rating
Do not show users with less than N games
I worked on a Scala implementation a few months ago, although it was a bit unpolished--I should go back to finish it. I at least got some reasonable results from it.

If you win every game you play, yes, your rating will grow to be quite high even if you play against only low rated players. The probability that you will win every game against such an opponent is probably about what a class A player would achieve (although it probably is a bit inflated due to the relatively high RD, still).

The best way to counter, in my opinion, is to not consider someone with an RD over a certain amount to be a stable rating--i.e. consider it to be "provisional". Also, in order to actually gain a title, at least in USCF, they have a system of norms, where you must perform at a certain level in a tournament of 4 games or more (4 times, I believe), which makes the likelihood of playing against a ~1378 for the whole tournament [four times] very unlikely.

Is your goal to use this for chess? What is your use case?

Update:
FICS handles it by only considering people with an RD of &lt;80 active. (They use Glicko-1, still, I believe.) http://www.freechess.org/Help/ficsfaq.html#Q005.003

And by the way, Glicko-1 uses RD/time decay as well. Glicko-2's main improvement was the "volatility" factor, which allows people with erratic results or stable results to be calculated very slightly differently. I think it's a very minor tweak to Glicko-1 which causes considerable extra calculation--but like you, I was still interested in calculating it. I actually asked Glickman himself for some additional datapoints for testing, but he was too busy to supply them at the time.