According to the Laws of Chess:
7.4.1 If a player displaces one or more pieces, he shall re-establish the correct position in his own time.
Of course, in this situation, it is not the player who displaced one or more pieces. I believe that another rule would therefore be in effect:
7.6 If, during a game it is found that any piece has been displaced from its correct square, the position before the irregularity shall be reinstated. If the position immediately before the irregularity cannot be determined, the game shall continue from the last identifiable position prior to the irregularity. The game shall then continue from this reinstated position.
Note the use of passive voice. If 7.4.1 doesn't apply, then it doesn't matter how it got displaced and it doesn't matter how it gets put back, but the position shall be reinstated, and the game shall continue from the reinstated position. This implies that you can't move until it is reinstated (otherwise you wouldn't be continuing from that position.)
And, of course, you're allowed to stop the clock and call the arbiter if such a thing happens.
7.4.2 If necessary, either the player or his opponent shall stop the chessclock and ask for the arbiter’s assistance.