13

NCM data

If the NCM data is to be believed, somewhere around October 2020, Stockfish dev hit +370 elo vs. Stockfish 7, but has since regressed to only +360. What's going on?

Only thing I can think of is a statistical fluctuation, which is plausible since the error bars are +/- 5. However, the gap between the October ratings and the current ones appear to be too large to fully attribute to statistics. This is especially the case if we believe Stockfish improved between October and December (which it presumably must have since engines are constantly improving). On the other hand, if the October versions of Stockfish are truly stronger than the latest ones, the newer versions ought not to have been accepted.

Improve this question

1 Answer 1

Reset to default
21

It is of course always possible that Stockfish did regress. The way changes are tested for Stockfish, or most engines for that matter, always allows for regressions to pass some of the time. (you simply can't play long enough test matches in reasonable time to be very sure that you don't regress) This is not much of an issue since most of the time you will detect which changes brought improvement and which did not. However you can "run unlucky" and hit a couple not so good changes in a row. In fact I do recall an instant when Stockfish was reverted a number of commits since such a regression was measured.

However, in this case if the tests were run against Stockfish 7. I think a more likely explanation may simply be that the newer Stockfish did slightly worse against that specific opponent. Strength is not absolute, you can be better against some opponents and worse against others. The interesting thing then of course is to be on average stronger than the older version however that can still allow an engine to be weaker against certain opponents. Especially, if like the case here, the opponent is several hundred points weaker. At that point it is less about playing the absolute best move and more about beating up the much weaker opponent more efficiently. And especially since the supposed regression is only 10 rating points, I wouldn't interpret too much into it.

Improve this answer
3
  • 2
    Another possible cause is that SF recently has been getting further away from it's pre NNUE version, which may be removing a previous self-elo effect.
    – Oscar Smith
    Dec 15, 2020 at 4:47
  • @OscarSmith oh hey, some Leela folks here? :D Sounds very plausible, I wonder if I should edit it into my answer. But having it in the comment below may be good enough.
    – koedem
    Dec 15, 2020 at 4:57
  • I left it as a comment because imo, it's too speculative for me to put it as an answer. Your answer of hard to say is definitely right.
    – Oscar Smith
    Dec 15, 2020 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.