Theoretically, why is 4. Bd3 NOT one of the best candidates in Caro-Kann? I seem to find 4. Bd3 to be one of the best choices in Caro-Kann.

[FEN ""]

1. e4 c6 2. d4 d5 3. e5 Bf5 4. Bd3


My reasoning: 


White's light bishop can only develop to Be2 or Bd3. Be2 seems passive.
In Caro-Kann, Black gives up the center initially to attack it later (say by c6-c5 or other means), thus losing tempo. One of the reasons given is that Black gets the advantage of its light bishop being outside the pawn chain. By immediately challenging it, White seems to attack Black's advantage.
At move 4, Black's most active piece is its light bishop. Immediate 4. Bd3 either forces exchange thus bringing Black to square one in development; or it gets Black to lose another tempo with Bf5-Bg6.
At 4. Bf3, Black doesn't have the good option of Ne7-Nxf5, which becomes a possibility if White plays Bf3 later.


But I have never seen 4. Bd3 played in titled games.

Can the experts here help me qualitatively understand what is wrong with my reasoning?
The line usually quoted is 4.Bd3 Bxd3 5.Qxd3 Qa5+ 6. Bd2 Qa6!  If White now exchanges Queens or allows the Queens to be exchanged he already has a poor endgame structure with a bad dark-square Bishop. Otherwise he will have difficulty Castling. Certainly White is not lost, but he has given away his first-move advantage.

[FEN ""]

1. e4 c6 2. d4 d5 3. e5 Bf5 4. Bd3 Bxd3 5. Qxd3 Qa5+ 6. Bd2 Qa6!