2

I'm basically fishing for suggestions here... there is an AI contest to figure out a person's rating from a single game. The contest's description mentions Ken Regan's approach that makes use of engine evaluations of players' moves (and the contest's accompanying game data is pre-equipped with engine evaluations for such use). What other kinds of factors do you believe would be useful for such a task?

I could imagine things like:

  1. Ability to follow recent opening theory (for strong players)
  2. Total number of blunders (for weak players)
  3. At what point in the game did resignation occur? (for strong players, it often evaluates to numbers closer to a draw than for weak players)
  4. Average depth of tactical traps avoided

Other ideas?

  • 4
  • 3
    Short answer: You can‘t. – Keba Jan 7 '15 at 0:36
  • So I assume there would be a human with a known rating that is kept secret. The AIs would play a game then estimate the human's rating? Or, two players with secret ratings would play, and the AIs would attempt to estimate each?? – Tony Ennis Jan 7 '15 at 0:57
  • 1
    @DagOskarMadsen, while I agree that this question (especially with its title) looks very much like that earlier one in many respects, I don't believe it is a duplicate. In particular, the earlier question asked for software that could do such a thing, and its accepted answer was about Ken Regan's algorithm based on engine evaluations. Here, tbischel's question links to a kaggle competition whose description already mentions Regan's approach and whose dataset comes equipped with engine evaluations for all moves. (cont'd) – ETD Jan 7 '15 at 2:22
  • 1
    (cont'd) I see the present question as essentially asking how one can go beyond Regan's work, e.g. by complementing its approach with additional considerations, in order to do a better job of predicting Elo ratings. Perhaps it could use some rewording, but at least as I read it, it isn't a duplicate. (@tbischel, I have edited the question along the lines I described, but if I have altered your intent at all, please roll it back.) – ETD Jan 7 '15 at 2:24
3

I also try to take part in that competition, although I'm not sure I will be able to spare the time and computational effort to actually make a submission.

Unfortunately I think the Kaggle data set of engine evaluations isn't suitable to get the best possible result. I think it is crucial to not only have the evaluation of the best move, but also of the inferior choices. The reason is that you need to calculate some kind of measure of the complexity of the position to weight the accuracy shown by the players.

In short: It is hard to be accurate in a complex position and easy to be accurate in a simple position.

Another problem is that at a runtime of one second per move, stockfish probably isn't much stronger than the average player in the dataset (which would be around 2200 Elo) …

So to create a decent dataset one would have to invest a lot of cpu-hours. 50000 games x 40 moves x 1second (or more) >= 555 hours.

To create an opening tree that allows you to see when the players deviate from already played games and how strong the players of the played games were, is another idea to give an initial guess of the players strength.

So my rough outline is to create an opening tree to get an initial guess of the strength, ignore the moves played in the opening, weight the accuracy of the moves after the opening phase with the complexity of the position and do some kind of linear regression analysis. Maybe look out not to overvalue time trouble blunders.

I think the key is to get a good "complexity number". The other possibilities you mention … "number of blunders" just reduces "accuracy of play" to something similar with less information. "Depth of tactical traps avoided" is tricky and may computationally be even more expensive.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.