The relevant part in FIDE's Laws of Chess is Article 4.3 which contains the word deliberately.
4.3 Except as provided in Article 4.2, if the player having the move deliberately touches on the chessboard:
a. one or more of his own pieces, he must move the first piece touched which can be moved
b. one or more of his opponent’s pieces, he must capture the first piece touched which can be captured
c. one piece of each colour, he must capture the opponent’s piece with his piece or, if this is illegal, move or capture the first piece touched which can be moved or captured. If it is unclear, whether the player’s own piece or his opponent’s was touched first, the player’s own piece shall be considered to have been touched before his opponent’s.
The situation you describe once happened to a team mate of mine. The opponent wanted him to move the touched piece according to the rule, but the arbiter explained it had to be deliberate.