I'm going to further expand a FEN validation function I have been working on for some time to include illegal pawn formations such that:
... for example white pawns could never be on a2, a3, and b2; there is no way a pawn could be on both a3 and b2. (source)
This particular test doesn't require to take opponent missing material into consideration (it could, but I have that on a separate test to check every column with multiple pawns and determine the minimum missing material the opponent should have, but that test is different, I only want to focus on this other test assuming infinite material can be used to reach those positions).
So far, my approach has been to draw a triangle from any pawn (excluding the ones that sit on the starting rank, which wouldn't allow a triangle to be drawn) all the way to the starting rank, then counting all the pawns that would fit on the base line of the triangle (starting rank). In the case the triangle goes out of bounds, those pawns are not counted:
That amount of pawns on the bottom of the triangle are the upper exclusive limit that pawns could then appear anywhere inside the triangle (because the test is applied to each pawn and each one will draw their own triangles and because the test is a generalized "no invalid pawn structures?", as in, all or nothing, it is safe to give big triangles with cluttered pawns a pass, since later on, the pawns inside will undergo the same test).
Example from the left image with limit of 3, there should be 0-2 to be a legal pawn structure.
Will this solution always work or there are times where this limit will be incorrectly generated? Can this be simplified further using a different reasoning or methods?
Any ideas, feedback, alternative solutions are appreciated.