No, this move (...Rf8#) is not legal. According to the FIDE Laws of Chess effective 1 January 2018:
3.10.2 A move is illegal when it fails to meet the relevant requirements of Articles 3.1 – 3.9
(This improves on pre-July 2014 rules by specifically defining “illegal” as well as “legal”.)
In this case, the relevant requirement is in Article 3.9.2:
No piece can be moved that will … leave [the king of the same colour] in check.
The consequence of this is that the checkmate does not count:
5.1.1 The game is won by the player who has checkmated his opponent’s king. This immediately ends the game, provided that the move producing the checkmate position was in accordance with Article 3 …
Instead, the game shall continue with a legal move (after reverting the illegal move):
7.5.1 An illegal move is completed once the player has pressed his clock. If during a game it is found that an illegal move has been completed, the position immediately before the irregularity shall be reinstated. … Article[] 4.3 … appl[ies] to the move replacing the illegal move. The game shall then continue from this reinstated position.
Article 4.3 is the so-called “touch-move” rule and, in this case, would require Black to play ...Rf2 (probably losing the rook on the next move).
Actually, the key point here is also covered in the introductory Article 1:
Leaving one’s own king under attack … is not allowed .
As an aside, ...Rb8# would also be checkmate, but would also be illegal. This rook has no legal moves, so “touch-move” would not be relevant here and Black would be able to make any legal move after the illegal move was reverted.