Let's start at the beginning
To understand X-FEN for castling in Chess960, it helps if you first understand castling in Chess960. Castling rules are slightly different in Chess960, hence there comes a need for X-FEN. I believe it's the major source for your confusion.
Minor Note 1 X-FEN is used not just for Chess960 but for many other variants as well (Capablanca chess, Janus chess, Chess480, etc.)
Castling in Chess960
Castling in chess960 is done on two sides: the a-side and the h-side, analogous to queenside and kingside castling respectively. The final positions after castling are the same in Chess960 and in standard chess.
The terms a-side and h-side are used as the initial board configuration is randomized, and what is queenside in normal chess needn't be so in Chess960 (the same applies for kingside.).
Why FEN is insufficient for Chess960
FEN is sufficient for standard chess, as there are only two initial squares for the rooks, on the a-file and the h-file.If a rook from the kingside somehow gets to say, the c-file, there is no question of allowing queenside castling with it as it isn't on the right square. (More on this in an example.)
In chess960, the rooks have many possible initial squares. In the course of a game, both the rooks could end up on the same side of the king. Now, if FEN is used, there is no way of clearly knowing which rook has the castling rights (without knowing the initial position) as either could be occupying the legal square for castling. As a matter of fact, X-FEN assumes by default the outermost rook in such cases to have the castling rights, but that needn't be the case always.
An example to make my point clear
If the above position occurred in a standard chess game, the FEN would simply be
rn2k1r1/ppp1pp1p/3p2p1/5bn1/P7/2N2B2/1PPPPP2/2BNK1RR w Kq - 4 11
This is because White can castle legally on the kingside, with the rook on the h-file. The rook on the g-file would play no role in castling, simply because that isn't on the legal square for kingside castling.
Likewise, Black can castle only on the queenside, hence the FEN has a q for the castling rights of Black. The rook on g8 is irrelevant.
Minor Note 2 Precise readers might point out that the number of moves and half-moves required to arrive at the position shown, in a normal chess game, might not tally with the values given in the FEN. That is quite possibly true, but I stuck to the value given for the Chess960 game (for the same position) on Wikipedia, for simplicity's sake.
The situation changes if the game is Chess960, simply because the legal initial square for the white rook could be either g1 or h1. We cannot be sure without knowing the initial position. We cannot ascertain which rook has the kingside castling rights.
If the FEN given above was used here, it would confer the castling rights on the rook on the h-file, as the outermost rook is assumed to be the one with the castling rights, by default. It would also mean that Black can castle only on the a-side (queenside, in other words).
As a matter of fact, the initial position for this game was actually
rnbnkqrb/pppppppp/8/8/8/8/PPPPPPPP/RNBNKQRB w KQkq -
If you notice the bold R's, you'll see that the legal square for the white kingside rook (the one involved in h-side castling) is actually g1, not h1. Similarly, the home square of the black 'kingside' rook is g8, not h8.
This means that using normal FEN not only confers the castling rights to the wrong white rook, it also fails to indicate the ability of Black to castle kingside. It's because normal FEN assumes the initial squares of the rooks to be on the a- and h-files only.
Hence, to avoid the ambiguity caused by normal FEN, X-FEN records the position as
rn2k1r1/ppp1pp1p/3p2p1/5bn1/P7/2N2B2/1PPPPP2/2BNK1RR w Gkq - 4 11
The G (in bold) shows that the castling rights on the kingside are with the rook on the g-file, not with that on the h-file. This is the inner rook. Similarly, it ensures that Black's castling rights on the kingside aren't ignored.
The queenside castling possibility for Black is recorded normally as there is no ambiguity.
Recording castling in Chess960 using X-FEN works as follows:
Just like normal FEN, if the two rooks are on the opposite sides of the king.
Still like normal FEN, if both rooks are on the same side of the king, with the outermost rook having the right to castle.
Using the letter (capital for White, small for Black) of the file of the rook permitted to castle, when both rooks are on the same side of the king, with the innermost rook having the right to castle. In the above example, G is used to indicate that the inner rook of White (on the g-file) has the right to castle.
Hope that's clear, feel free to ask if there's any confusion!
Note: Special case
In the unusual scenario of three or more rooks (due to underpromotion) being on the same side of the king, with the king yet to castle, there would be an outermost rook, an innermost rook and one or more rooks in between. In such a case, the outermost rook is treated the same way as above, and any other inner rook is identified by its file (depending on which has the right to castle).
For the curious, here's the PGN of the entire example game.
[Event "SmirfGUI Computerchess Game"] [Site "CHESSBOX"] [Date "2005.06.19"] [Time "10:22:29"] [Round "Test"] [White "White"] [Black "Black"] [Result "*"] [Annotator "R. Scharnagl"] [SetUp "1"] [FEN "rnbnkqrb/pppppppp/8/8/8/8/PPPPPPPP/RNBNKQRB w KQkq - 0 1"] 1. h4 g6 2. g3 Bf6 3. a4 Qh6 4. Ra3 Bxh4 5. gxh4 Qxh4 6. Qh3 Qxh3 7. Rxh3 Ne6 8. Bf3 d6 9. Nbc3 Ng5 10. Rhh1 Bf5 11. O-O *
Image and PGN courtesy Wikipedia