OK, so it's your turn and you're in the middle of a deep long think in a complicated position when this big hairy arm reaches all the way across the board obscuring your view of a large part of the board and adjusts your castled king and rook. It's not just a quick tap of the piece. It's a lengthy, thumb and 2 fingers adjusting of the pieces. That's a good thing in your view? It doesn't matter, by the way, whether your opponent says "j'adoube" or anything else in that situation. It's not his turn and so he can have no intention of moving the pieces touched.
Do that to me and I'll immediately call the arbiter and I'll expect him to penalize you by giving me an extra 2 minutes on the clock. If an arbiter is actually watching and sees it happen I don't expect to have to complain and tell him. I expect him to intervene immediately and penalize you.
On the other hand we have in our club a couple of players with the following bad habit. They make a move, press the clock and then instinctively touch the piece further on its square whether it needs it or not before writing the move down. At standard time controls where I'm also writing the moves down I'd probably let this go but at rapid or blitz when time is more critical I'd complain to the arbiter.
Note that when I'm arbiter in a competition with these two players I warn them after the game if they do it and their opponent doesn't complain. If an opponent complains then I give an automatic 2 minute penalty. One player is pushing 70 and I don't really expect him to change. It's only thanks to the increment that he no longer hits the clock in Zeitnot, but the other player is in his mid 20's and quite strong (2100+). He needs to fix this bad habit before he starts getting penalized in stronger competitions.