Can you take back a draw offer? Is it possible to take back the draw offer if the opponent was not immediately replying to it?
No. Rule 9.1.2.1 of the FIDE rules says it all:


  9.1.2.1 A player wishing to offer a draw shall do so after having made a move on the chessboard and before pressing his clock. An offer
  at any other time during play is still valid but Article 11.5 must be
  considered. No conditions can be attached to the offer. In both cases
  the offer cannot be withdrawn and remains valid until the opponent
  accepts it, rejects it orally, rejects it by touching a piece with the
  intention of moving or capturing it, or the game is concluded in some
  other way.


(rule 11.5 is about not distracting the opponent during his move)

It's expected that the opponent takes his time to think about the offer, just as he probably spends time on thinking about his next move. He isn't required to actually refuse the offer, just making a move is quite normal too (once he touches a piece to move the offer is over, see above).