The answer depends on whether or not the games are being recorded. The way this is elliptically referenced in the FIDE Laws of Chess is that a distinction is made between, on the one hand, standard time controls (where moves must be recorded by the players), rapid and blitz games played at a sufficiently high level that there are enough arbiters to record the games and on the other hand regular rapid and blitz where nobody is recording the rules.
In the first case article 7.5.1 applies and the games is restarted from the last known legal position and various penalties can apply.
7.5.1 An illegal move is completed once the player has pressed his clock. If during a game it is found that an illegal move has been completed, the position immediately before the irregularity shall be reinstated. If the position immediately before the irregularity cannot be determined, the game shall continue from the last identifiable position prior to the irregularity. Articles 4.3 and 4.7 apply to the move replacing the illegal move. The game shall then continue from this reinstated position.
According to Appendix A 7.5.1 also applies for rapid where:
A.3.1.1 one arbiter supervises at most three games and
A.3.1.2 each game is recorded by the arbiter or his assistant and, if possible, by electronic means.
And for blitz according to Appendix B where
B.3.1.1 one arbiter supervises one game and
B.3.1.2 each game is recorded by the arbiter or his assistant and, if possible, by electronic means
Otherwise for rapid and blitz (recording not done / not possible) illegal moves are not punished and not corrected unless either the arbiter or one of the players spots what has happened before another move has been made:
A.4.2 If the arbiter observes an action taken under Article 7.5.1, 7.5.2, 7.5.3 or 7.5.4, he shall act according to Article 7.5.5, provided the opponent has not made his next move. If the arbiter does not intervene, the opponent is entitled to claim, provided the opponent has not made his next move. If the opponent does not claim and the arbiter does not intervene, the illegal move shall stand and the game shall continue. Once the opponent has made his next move, an illegal move cannot be corrected unless this is agreed by the players without intervention of the arbiter.
There is an additional twist for rapid and blitz to handle the position where the arbiter comes late to the scene and spots your last scenario:
my opponent moves a piece on the other side of the board leaving the kings checking each other and hits the clock, what happens here?
The arbiter waits one move to see if the illegal position is corrected and if not declares a draw.
A.4.4 If the arbiter observes both kings are in check, or a pawn on the rank furthest from its starting position, he shall wait until the next move is completed. Then, if an illegal position is still on the board, he shall declare the game drawn
If the arbiter doesn't see it then the game between Larry and Mo continues.