One way out of this would be to invoke article 3.9.2 of the FIDE Laws:
No piece can be moved that will either expose the king of the same colour to check or leave that king in check.
Yep, that seems entirely unambiguous to me. You can't leave your king in check after your move. (Your objection seems to be predicated on the idea that a move that touches two pieces can be said in some sense to have touched no pieces, and I think that's just wrong. You might with more grounding claim that article 3.9.2 permits you to end your turn in check if you got there by moving a pawn, since a pawn is "no piece"! :))
FIDE's Chess960 rules also state explicitly that when you castle, usually it's by moving two pieces one after the other:
II.3.2.1 double-move castling: by making a move with the king and a move with the rook, or
II.3.2.2 transposition castling: by transposing the position of the king and the rook, or
II.3.2.3 king-move-only castling: by making only a move with the king, or
II.3.2.4 rook-move-only castling: by making only a move with the rook.
So in your particular example, it's the "move with the rook" (II.3.2.1) that exposes its king to check (3.9.2) and is thus disallowed.
However, I admit that if you use that wording as gospel, then you have to point out that "transposing the position of the king and the rook" doesn't literally "move" either of the pieces; it just "transposes" them without using the word "move." So you could still ask your question about
4k3/8/8/8/8/8/8/r1RK4 w Q - 0 1
In that case, we'd just have to fall back on common sense: that you can't move your king into check, period. Notice that David Wheeler states that "the king may not be in check after castling" falls naturally out of the FIDE rules, without feeling the need to justify it any further:
[...] castling may only occur under the following conditions, which are extensions of the standard rules for castling:
- Unmoved: The king and the castling rook must not have moved before in the game, including castling.
- Unattacked: All of the squares between the king's initial and final squares (including the initial and final squares) must not be under attack by any opposing piece.
- Vacant: All the squares between the king's initial and final squares (including the final square), and all of the squares between the rook's initial and final squares (including the final square), must be vacant except for the king and castling rook.
These rules have the following consequences: [...]
- The king may not be in check before or after castling.
For what seem like "better" rules of castling, see John Kipling Lewis's "Castling in Chess960: An appeal for simplicity" (2005).