The main reasons it is OK for black is that he is still down only one tempo in piece development, but he has traded off his bad bishop for white's good bishop, and his position is still very solid so he will catch up in development eventually. The downside is that white has more space. Black can eventually fight back with c5 after finishing his development, which is the key to the opening from black's perspective.
I have played the Caro for decades, and it is very solid, but you have to be very patient, and well-prepared, or it is easy to get rolled over due to white's space.
Here are a few examples from the highest level. The first, is black successfully hitting back with c5, breaking up the white center. The second and third are Kasparov showing how white can successfully use the kingside pawns (I am posting these primarily to show that my comment below about h5 not really being weak is accurate).
[Event "Corsica Masters 5th"]
[Site "Bastia"]
[Date "2001.10.30"]
[Round "2.2"]
[White "Leko, Peter"]
[Black "Anand, Viswanathan"]
[Result "1/2-1/2"]
[ECO "B19"]
[WhiteElo "2739"]
[BlackElo "2770"]
[FEN ""]
1. e4 c6 2. d4 d5 3. Nc3 dxe4 4. Nxe4 Bf5 5. Ng3 Bg6 6. h4 h6 7. Nf3 Nd7 8. h5 Bh7 9. Bd3 Bxd3 10. Qxd3 e6 11. Bf4 Bb4+ 12. c3 Be7 13. O-O-O Ngf6 14. Kb1 O-O 15. Ne5 c5! {breaking up white's center} 16. Ne4 cxd4 17. Qxd4 Nb6 18. Nxf6+ Bxf6 19. Qe4 Qe7 20. Ng4 Rfd8 21. Nxf6+ Qxf6 22. Be3 Rxd1+ 23. Rxd1 Nd5 24. Bd4 Qf4 25. Qxf4 Nxf4 26. g4 b5 27. Kc2 Nd5 28. Kb3 a6 29. a4 f6 30. Ra1 Rb8 31. axb5 axb5 32. Ra7 e5 33. Rd7 exd4 34. Rxd5 dxc3 35. bxc3 g6 36. hxg6 Kg7 37. Kb4 Kxg6 38. Rxb5 Re8 39. c4 Re4 40. f3 Rf4 41. Rd5 Kf7 42. Rd3 f5 43. gxf5 h5 44. Kb5 h4 45. c5 Rxf5 46. f4 Rh5 47. Rh3 Ke6 48. Kb6 Rh6 49. c6 Kd5 50. Rd3+ Ke4 51. Rh3 Kxf4 52. Kb7 1/2-1/2
The next game shows the structure I mentioned in the comment below. Kasparov did not actually break through on the kingside, but he used the cramping threats on the kingside to eventually attack the queenside! Anand might have been able to draw in a couple of places, but that is no easy feat against the great Kasparov. A beautiful rook ending by Kasparov.
[Event "Linares 20th"]
[Site "Linares"]
[Date "2003.03.01"]
[Round "7"]
[White "Kasparov, Garry"]
[Black "Anand, Viswanathan"]
[Result "1-0"]
[ECO "B19"]
[WhiteElo "2847"]
[BlackElo "2753"]
[FEN ""]
1. e4 c6 2. d4 d5 3. Nd2 dxe4 4. Nxe4 Bf5 5. Ng3 Bg6 6. h4 h6 7. Nf3 Nd7 8. h5 Bh7 9. Bd3 Bxd3 10. Qxd3 Ngf6 11. Bf4 e6 12. O-O-O Be7 13. Kb1 O-O 14. Ne4 Nxe4 15. Qxe4 Nf6 16. Qe2 Qd5 17. Ne5 Qe4 18. Qxe4 Nxe4 19. Rhe1 Nf6 20. g4 Rfd8 21. Be3 Bd6 22. f3 Rac8 23. c4 a5 24. a4 Nd7 25. Bd2 Bc7 26. Bc3 Nxe5 27. dxe5 c5 28. Kc2 Rxd1 29. Kxd1 Rd8+ 30. Ke2 Rd7 31. f4 Bd8 32. f5 Bg5 33. f6 b6 34. Ra1 Rd8 35. Ra3 gxf6 36. Rb3 Bf4 37. Rxb6 Bxe5 38. Bxe5 fxe5 39. Rb5 Rd4 40. Rxc5 Rxg4 41. b3 Rg3 42. Rxe5 Rxb3 43. Rxa5 f5 44. Ra8+ Kg7 45. c5 Rc3 46. Rc8 Ra3 47. c6 Rxa4 48. Re8 Rc4 49. Rxe6 f4 50. Rg6+ Kh7 51. Kd3 Rc5 52. Kd4 Rc1 53. Ke4 Rc4+ 54. Kd5 Rc3 55. Kd4 Rc1 56. Ke4 Rc4+ 57. Kf3 Kh8 58. Rxh6+ Kg7 59. Rd6 Kh7 60. Kg4 Kg7 61. Rd7+ Kf6 62. c7 1-0
Lastly, here is a game that made an big impression on me when I first read Kasparov's notes to in in his book, "The Test of Time". How he used the kingside pawns was amazing, and his final move shows how dangerous they are. I will use a few of Kasparov's own annotations for the demonstrative points about the pawn play.
[Event "EU Team-ch07 Final"]
[Site "Skara"]
[Date "1980.01.26"]
[Round "7.8"]
[White "Kasparov, Garry"]
[Black "Vukic, Milan"]
[Result "1-0"]
[ECO "B19"]
[WhiteElo "2595"]
[BlackElo "2460"]
[FEN ""]
1. e4 c6 2. d4 d5 3. Nd2 dxe4 4. Nxe4 Bf5 5. Ng3 Bg6 6. h4 h6 7. Nf3 Nd7 8. h5 Bh7 9. Bd3 Bxd3 10. Qxd3 e6 11. Bd2 Qc7 12. O-O-O Ngf6 13. Ne4 O-O-O 14. g3 Nxe4 15. Qxe4 Be7 16. Kb1 Rhe8 17. Qe2 Bd6 18. Rhe1 Re7 19. c4 c5 20. Bc3 Nf6 21. Ne5 cxd4 22. Rxd4 Bxe5 23. Rxd8+ Qxd8 24. Bxe5 Rd7 25. Bc3 Qb6 26. g4 Qd6 27. f3 a6 28. a4 Qd3+ 29. Kc1 Kc7 30. Qxd3 Rxd3 31. Rf1 Kc6 32. Kc2 Rd7 33. a5! {+- "the ending, in my opinion, is lost for Black - he is markedly cramped, and the balance of forces remaining on the board is the ideal one for White." - Kasparov} Ne8 34. Re1 Rd6 35. f4 Nf6? {"One can understand Vukic aiming to prevent g4-g5, after which the sacrifice at g7 becomes a possibility in various lines, e .g. 35...Kd7 3 6 g5 Rc6 37 Rdl + Kc8 38 BXg7! But, engaged in a difficult defence, the Yugoslav grandmaster forgot for an instant about the defects of his pawn structure and the unfortunate position of his rook at d6. The retribution followed immediately."} 36. Bxf6 gxf6 37. Rd1 {"Black resigns, since in the pawn ending White creates passed pawns on both flanks."} (37. Rd1 Rxd1 38. Kxd1 Kc5 39. g5 fxg5 40. fxg5 hxg5 41. h6 {wins}) 1-0
This final position, and the ability to create a passed pawn on the king side, combined with the pawn move f6 in the second game to do the same (after gf ef and later g5) is why this structure is very strong for white.