A part of it has to be that white doesn't have to keep the pawn.
[FEN ""]
1. d4 e5 2. dxe5 Nc6 3. Nf3 Qe7 4. Nc3 Nxe5
4.Nc3
is actually the most popular move in lichess database. Black Queen is placed a bit strangely on e7, it is blocking the development of the dark-squared bishop and not supporting the d-pawn if it wants to move to d5. Black can double white pawns with Nxf3, but white has plenty of activity to compensate for it. White has many promising continuations, but here is one:
[FEN ""]
1. d4 e5 2. dxe5 Nc6 3. Nf3 Qe7 4. Nc3 Nxe5
5. Bf4 Nxf3+ 6. gxf3 Qb4 7. Qd2 d6 8. e4
This position is not exactly a gambiter's dream, since white is much more active, and also stockfish gives it 1.5.