I remember my first steps when starting out with the game.
I was playing against a neighbor of mine who was much much stronger at that moment.
He used all of his units better than me, including knights.
In order to cure the phobia you currently have you must do the following:
You must learn your openings well, so you do not allow knight on b4/g4 to be a problem.
Knight jump on b4/g4 is not a good move in general, and only in some openings it can "pass" but as a defensive move.
As someone who has "been there", I advise you to just learn your openings very well, and you will be fine.
You can start with Starting Out-... books from Everyman Chess since they are suitable for beginner/intermediate level, and usually in them you can find some very good general advice related for other parts of the game.
Hopefully this will help you.
Best regards.