According to the 2018 FIDE Laws of Chess (emphasis mine):
4.2.1 Only the player having the move may adjust one or more pieces on their squares,
provided that he first expresses his intention (for example by saying “j’adoube” or “I
4.2.2 Any other physical contact with a piece, except for clearly accidental contact, shall be considered to be intent.
4.3 Except as provided in Article 4.2, if the player having the move touches on the
chessboard, with the intention of moving or capturing:
4.3.1 one or more of his own pieces, he must move the first piece touched that can be moved
Touching with a forearm seems to be "clearly accidental contact", so it shouldn't trigger the touch move rules.
In the scenario where you move the piece because your opponent said you must, and then want to take it back, it's less clear. You generally cannot take back moves (and, if you've moved, then obviously you have now touched the piece with the intent of moving it.) However, your opponent has acted inappropriately; at the very least they've distracted you during your move. It's possible the arbiter would give some sort of penalty for this, especially if your opponent knew touch-move didn't really apply. You should summon the arbiter as soon as possible to get a ruling.