Why does NCM show no progress for Stockfish when each new version has to beat the previous version?

Question

NCM plays each Stockfish dev build 20,000 times against Stockfish 7. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds. As of time of writing, one can see from the plot that Stockfish's strength hasn't really budged for a month: version 20190324-1637 had a performance of +217.95 +/- 3.73 elo relative to Stockfish 7, while the latest version 20190420-0634 is +221.52 +/- 3.79 elo. The gap is within the error bars.

(It's possible that this is sort of the expected performance increase in a month's time, but the latest version's results appear to be anomalously high. For example the next-latest version, 20190419-1533, is +218.61 +/- 3.77 and almost indistinguishable from the 20190324-1637 version.)

The weird thing is, for each version of Stockfish to be promoted to the "main version", it has to beat the previous version. All these new versions of Stockfish (that aren't non-functional changes) have to do this. If we look at the catalogue of Stockfish patches, we see patches such as:

Author: protonspring Date: Tue Apr 16 23:10:53 2019 +0200 Timestamp: 1555449053

Move pawnsOnSquares to Position (#2100)

We can remove the values in Pawns if we just use the piece arrays in Position. This reduces the size of a pawn entry. This simplification passed individually, and in concert with ps_passedcount100 (removes passedCount storage in pawns.).

STC LLR: 2.95 (-2.94,2.94) [-3.00,1.00] Total: 19957 W: 4529 L: 4404 D: 11024 Elo +2.18 http://tests.stockfishchess.org/tests/view/5cb3c2d00ebc5925cf016f0d

Combo STC LLR: 2.95 (-2.94,2.94) [-3.00,1.00] Total: 17368 W: 3925 L: 3795 D: 9648 Elo +2.60 http://tests.stockfishchess.org/tests/view/5cb3d3510ebc5925cf01709a

In other words this version of Stockfish defeated the previous one by ~2 elo. Why isn't this ~2 elo showing up in the NCM data? I know elo isn't transitive (i.e. if engine A beats engine B by 50 elo and engine B beats engine C by 50 elo, it doesn't mean engine A beats engine C by 100 elo), but this is just one patch, and there were plenty of elo-gaining patches that passed Fishtest. For NCM to show no progress for months, in spite of all these patches, doesn't make sense to me.

Answer 1

Fishtest tests proposed changes to Stockfish's code with SPRT to determine if they should be accepted. Broadly speaking, there are two types of changes which are tested in this way (non-functional changes do not go through Fishtest): simplifications, and changes which intend to gain strength - elo changes.

The crucial difference here is that simplifications (like the one referenced in the question) are tested for no regression, whereas elo changes are tested for an improvement: SPRT bounds of [0.5, 4.5] are used for elo changes and bounds of [-3, 1] are used for simplifications.

The vast majority of changes to Stockfish recently have been simplifications, and as such they are not expected to increase strength.

For example, the patch in the question is a simplification, and was tested with SPRT bounds [-3.00, 1.00]. The fact that it passed tells us that there is a 95% chance that the change does not lose 3 elo. If the test failed, we would know that there is a 95% chance that the change does not gain 1 elo.

We can't say anything more precise than this, because SPRT stops immediately as soon as there is enough evidence to accept either hypothesis and doesn't attempt to measure the elo change more precisely. Although an elo difference of +2.18 is quoted at STC, all we know is that the lower error bound on the elo gain is >= -3.00.

There is insufficient evidence to say that the change produced an actual elo gain of +2.18, and thus claiming that this simplification gains 2 elo would be misleading.

The small number of elo gaining changes is what results in Stockfish's slow progress at the moment.