Assume an observer sees a chessboard in the middle of a game. At which point in a game does it become impossible to determine who has to play next between White and Black?
As soon as triangulation becomes possible, which occurs as soon as anyone makes a move that allows a piece besides a knight to advance to the third rank, you cannot determine with certainty who has the move.
For example, with a pawn on e3, it could be Black to move because White just played their first move, or it could be White to move with a sequence like this:
[FEN ""] 1. e3 Nf6 2. Bc4 Ng8 3. Be2 Nf6 4. Bf1 Ng8 *
If any piece or pawn is captured (except the queens!) that also means you cannot determine who is to move:
[FEN ""] 1. Nc3 (1. Na3 Nc6 2. Nc4 Rb8 3. Nb6 Nb4 4. Na8 Rxa8 5. Nf3 Na6 6. Ng1 Nb8) 1...Nc6 2. Nd5 Nb8 3. Nb4 Nc6 4. Na6 Nb4 5. Nb8 Rxb8 6. Nf3 Ra8 7. Ng1 Na6 8. Nf3 Nb8 9. Ng1 *
But if no pawn has moved (or if the only pawn moves are a3/a6/h3/h6) and nothing has been captured (with the exception of the queens) then you can tell from the board position who has the move. The knights can never lose a tempo, and the rooks cannot triangulate if they don't have enough room to move more than one square at a time. The king also cannot triangulate if the only square available is the queen's square.