If you are trying to remove the pawn to cause the most damage to the starting position, I'd go with the
f pawn. Since
f2/f7 is already a tender spot, as it is only covered by the King in the early parts of the game, it's a bit more likely to cause overall positional damage.
You could make the argument that removing the
e pawn could be worse, as they are important for central control. However, removing them does add a certain value, in that the pieces can develop more quickly from the starting position.
If you are trying to cause the least damage to your own position, definitely either the
h pawn. They have the least influence on the board, and there is something to be said for having that open file for the rook.
As to which one to remove...if you followed the above advice and removed the
f pawn of your opponent, I'd remove my own
a pawn, so that they don't feel comfortable castling on either side of the board.