According to the chess.com DB, 3.e4 is the third most popular reply in the Queen's Gambit Accepted - but why isn't it the most popular by a long shot? Isn't the entire point to remove black's central d-pawn so white can play a strong e2-e4? The moves 3.Nf3 and 3.e3 seem passive almost, but surely I am missing something. Maybe e2-e4 is taking up too much space too early, and you end up overextending yourself?
I'm not sure which database you are looking at: I see 3.e4 as the second, not third, most popular choice on chess.com, though not by much:
Here's what one author says about 3.Nf3:
Throughout the history of the Queen’s Gambit Accepted, this has been White’s most popular move in this position. White does not hurry to win back the pawn that he has sacrificed. He calmly develops his pieces and offers his opponent as few targets for an immediate counterattack as possible.
Sterren, Paul Van der. FCO: Fundamental Chess Openings (Kindle Locations 1487-1489). Gambit Publications. Kindle Edition.
and about 3.e4:
For a long time this very natural reaction to 2... dxc4 suffered the same fate as 3 e3 and 3 Nc3: it was thought to be inaccurate because of 3... e5. The difference is that in this case White has the promising pawn sacrifice 4 Nf3! exd4 5 Bxc4!, when his development is swift and aggressive and Black will have to take great care in getting his king into safety. This is in fact a lively variation with chances for both sides.
Another standard device for counterattacking against the white centre consists in 3... c5.
Sterren, Paul Van der. FCO: Fundamental Chess Openings (Kindle Locations 1438-1441). Gambit Publications. Kindle Edition.
My take is that it is not that 3.e4 is bad (it actually has better statistical results for White!), but it is a matter of taste and more people prefer to play more solidly.
I play e3 because I prefer more positional play. I want to first improve my position before a possible e4. In a more closed game the loss of tempo is not very important. That said, e4 is perfectly playable. To me, this is a matter of style.
I guess that most players who would play 3. e4 would play 1. e4