Of course you must be ready to play the Caro-Kann if you answer 1. d4 with 1. ... c6. But that is a sound opening - so why not, if you already play it against 1. e4? You will probably be better prepared for the Caro-Kann than your opponent, who after all started with 1. d4. If I was a d4 player, I surely would play 2. c4, hoping to enter the Slav.
In that case, blacks most popular choice is 1. d5, but other moves are also possible. Black might play for a setup with d6, g6, Bg7, with or without Nf6.
There is also the interesting Malinoise defence, 1. d4 c6 2. c4 b5!?, which has been played by GM Rogers from Australia and IM Jadoul from Belgium. This should at least be a nice surprise weapon.
Playing 1. ... c6 might also be an idea against London system players, e.g. 1. d4 c6 2. Nf3 Qc7!? has been played a few times - not the best of course, but at least it is a nasty surprise for any white player who planned 3. Bf4. Might be worth a try against certain opponents (especially among London system players, there are some who don't know how to play without their beloved Bf4)
The accelerated London system 1. d4 c6 2. Bf4 could be answered with 2. ... Qb6!?, when white has to decide whether or not to sacrifice the pawn on b2.