Is it possible to take back the draw offer if the opponent was not immediately replying to it?
No. Rule 188.8.131.52 of the FIDE rules says it all:
184.108.40.206 A player wishing to offer a draw shall do so after having made a move on the chessboard and before pressing his clock. An offer at any other time during play is still valid but Article 11.5 must be considered. No conditions can be attached to the offer. In both cases the offer cannot be withdrawn and remains valid until the opponent accepts it, rejects it orally, rejects it by touching a piece with the intention of moving or capturing it, or the game is concluded in some other way.
(rule 11.5 is about not distracting the opponent during his move)
It's expected that the opponent takes his time to think about the offer, just as he probably spends time on thinking about his next move. He isn't required to actually refuse the offer, just making a move is quite normal too (once he touches a piece to move the offer is over, see above).