As others have pointed out, the points values themselves are merely an initial value to get you started. Over many many years, people have found that generally trading three pawns for a bishop is a fair trade.
The point values are just a tool to get you started. They're a way for you to not make mistakes by trading valuable pieces for less valuable pieces. But really, its the entire position that matters.
Consider this hypothetical. Your opponent has a pawn on the 7th rank. On the next turn, they can promote it to a queen. You have the potential to trade a rook for that pawn. Is it a good trade? The "5 points for a rook" versus "1 point for a pawn" fails to account for the position. In this intentionally extreme example, trading a rook for a pawn may be a great trade, because it puts you in a better position.
Indeed, it can be even more extreme. You can construct some clever back-rank mate problems where a pawn promoting to a queen or a rook is checkmate. If you can't get out of it otherwise, it would be worth sacrificing three, four, or even five whole pieces just to prevent that pawn from promoting. Checkmate trumps any and all point calculations, always.