Did Stockfish regress from October to December 2020? If so, why? NCM data
If the NCM data is to be believed, somewhere around October 2020, Stockfish dev hit +370 elo vs. Stockfish 7, but has since regressed to only +360. What's going on?
Only thing I can think of is a statistical fluctuation, which is plausible since the error bars are +/- 5. However, the gap between the October ratings and the current ones appear to be too large to fully attribute to statistics. This is especially the case if we believe Stockfish improved between October and December (which it presumably must have since engines are constantly improving). On the other hand, if the October versions of Stockfish are truly stronger than the latest ones, the newer versions ought not to have been accepted.
It is of course always possible that Stockfish did regress. The way changes are tested for Stockfish, or most engines for that matter, always allows for regressions to pass some of the time. (you simply can't play long enough test matches in reasonable time to be very sure that you don't regress) This is not much of an issue since most of the time you will detect which changes brought improvement and which did not. However you can &quot;run unlucky&quot; and hit a couple not so good changes in a row. In fact I do recall an instant when Stockfish was reverted a number of commits since such a regression was measured.
However, in this case if the tests were run against Stockfish 7. I think a more likely explanation may simply be that the newer Stockfish did slightly worse against that specific opponent. Strength is not absolute, you can be better against some opponents and worse against others. The interesting thing then of course is to be on average stronger than the older version however that can still allow an engine to be weaker against certain opponents. Especially, if like the case here, the opponent is several hundred points weaker. At that point it is less about playing the absolute best move and more about beating up the much weaker opponent more efficiently. And especially since the supposed regression is only 10 rating points, I wouldn't interpret too much into it.