To be concrete, I'll focus on the Olympiad; the same ideas apply more generally.
So, the more boards we have on each team, the more representative the match will truly be of the respective populations. Now 4 players is indeed a small sample, but I am curious, what would your response be to someone who might pose the further question:
"Why would a team championship have only 10 boards? That's not really representative of anything. I think that 20 boards would more appropriately represent the team's true strength."
We could of course push the same idea further and further. In principle, the only limiting factor for the number of boards would be the population of the smallest country participating, but of course things would become logistically impractical well before that.
Whatever you think of my turning your own question around at you, I think the gist of a simple answer lies in the last thing I said: different countries have vastly different sizes. If you want to have an elite team event that all countries can realistically compete in, you just can't have all that many boards. Russia would be happy to make it a 50-board competition, since they have a large population that is as chess-loving as any in the world. (According to the current live ratings at 2700chess, 12 of the top 43 players in the world are Russian.) But doing so would mean that far fewer countries have any real shot at doing at all well in the event; the caliber of player that, say, Israel would have to put on a Board 10 would be well below what Russia would be putting up.
Even with 4-person teams, only a handful of countries have done much medaling historically. But note that my example small country, Israel, got a silver medal in 2008 and a bronze in 2010. Things like that just wouldn't be possible with many more boards in play.