After:
23 ..f3 24 Nxf3 Bg4
You are threatening Bxf3 to win a piece, since the g2 pawn is pinned. So sacrificing the f3 pawn gives you great initiative and attacking options, and keeps white pinned down.
Even if white moves the king to avoid the pin, with say Kf1, you can take in f3 with the rook and if white captures back with the pawn Bh3 is mate:
[FEN "2br1rk1/p5pp/1p1p2q1/1PnRp1N1/2P1Pp1P/P7/2B1QPP1/4R1K1 b - - 0 1"]
1. f3 Nf3 2. Bg4 Kf1 3. Rf3 gxf3 4. Bh3#
Running the king to the other way, h1, is also equally bad, because Rf4 becomes undefendable:
[FEN "2br1rk1/p5pp/1p1p2q1/1PnRp1N1/2P1Pp1P/P7/2B1QPP1/4R1K1 b - - 0 1"]
1. f3 Nf3 2. Bg4 Kh1 3. Bxf3 gxf3 4. Rf4 Qf1 5. Rxh4 Qh3 6. Rxh3#
So as you can see, the attack becomes quite strong, so the engine suggests white to sacrifice the knight to gain counter attacking options, and not lose on the spot. Even though the position is still lost regardless.
I understand why eventually we arrive at the position below with 27..Be6. But the computer suggests 28.a4 instead of moving the rook
Why doesn't White, for example, exchange their rook with Black’s rook and then play 29. Qe2?
Well the exchange sacrifice on d5 isn't particularly bad considering the position, and how the d file becomes closed, but you get even further down material. Also other than Rxd8 you only have Rd6 to not sacrifice the exchange but that loses instantly to Rxd6 followed by Qxf2+ leading to mate.
The answer to "Why not Rxd8?" is: The engine didn't have enough time to think. The suggested move by Stockfish 10 in this position is in fact Rxd8 followed by either Qe3 or Qe2, as you can see in the following picture:
