Most of the time I see gambits are declined, but what are the advantages or disadvantages of accepting or declining gambits? I know this depends on the type of gambit, but I am looking for a more general answer that might apply to all gambits.
Since you are after general considerations:
Accepting a gambit. The advantage here of course is that you are gaining a material advantage, which, if you can hold onto it, provides well for your long-term prospects in the game. But what makes something an established gambit, rather than a mistake or a blunder, is that the offer of material is supposed to be offset by some sort of positional compensation, e.g. a lead in development, as in, say, the Danish Gambit:
[FEN ""] 1.e4 e5 2.d4 exd4 3.c3 dxc3 4.Bc4 cxb2 5.Bxb2
Or better control of the center in, say, the Queen's Gambit:
[FEN ""] 1.d4 d5 2.c4 dxc4 3.e4
Or open lines for pieces (like those bishops surveying two nice, long diagonals in the Danish Gambit).
So the potential disadvantage of accepting a given gambit is that you may be left with positional problems by which your opponent can punish you, perhaps by winning back the sacrificed material and then some, or worse, mating your king directly. Whether the compensation for your opponent outweighs the extra material you'll be getting is something that has to be decided on a case-by-case basis.
Declining a gambit. Here the main advantage, in my opinion at least, is simply that you're not giving your opponent the kind of position that he or she wants. For example, the Smith-Morra Gambit in the Sicilian Defense is very rarely played at the highest levels of the chess world, because the consensus among strong players is that white just doesn't get enough dynamic positional compensation (development lead, initiative) for the extra central pawn that black gets. But at lower levels, it can be much more dangerous to defend for black. Thus, if you're one to prefer slightly quieter positions rather than potentially needing to sweat under pressure for a while, one possible reaction to the Smith-Morra is to decline it by pushing d3 instead of capturing white's c pawn:
[FEN ""] 1.e4 c5 2.d4 cxd4 3.c3 dxc3 (3... d3 4.Bxd3) 4.Nxc3
Now white is not going to get the sort of development that was envisioned (for instance, the white pawn that now remains on c3 takes away the square which was intended for the knight).
The main disadvantage of this approach is obvious: no free material. And one shouldn't be afraid of ghosts in chess. If material is offered, one should certainly be cautious, but if at the end of your thinking you believe it is worth any trade-offs, then you should take it. Sometimes you will be wrong and get punished for your greed, but mistakes can be made at any juncture in the game, so this isn't a problem particular to gambit play. And besides, some gambits cannot be reasonably refused, e.g. the Cochrane Gambit in the Petroff Defense:
[FEN ""] 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nxf7
This is just like in many middlegames, where an attack will be launched with a sacrifice that cannot be refused, and one has no choice but to grab the material and hang on for dear life. So there is something to be said for not always declining aggressive gambits, because accepting can be good training for defending complicated positions, which aids one's overall development as a player.
In sum. As stated above, the decision as to whether the material offered in any given gambit is worth whatever positional trade-offs are involved is one that has to be made case-by-case. But hopefully I've given some useful general food for thought here.