i don't think that's weird at all. and i'm not a 'coordinate your outfit' kinda gal. i certainly don't worry about it if i'm just puttering around at home, whatever's handy. but if we're going out someplace and i know i'm going to be wearing a kid most of the time we're out, i don't want to horribly clash, nor do i want to wear an outfit that's going to look awful with a carrier on (like, say, a loose, flowing dress and a SSC). so if i really want to wear the dress, i'm more likely to grab a ring sling or a wrap. if i really want to wear the SSC, i'm more likely to pick pants and a fitted t-shirt. i do want babywearing to look good, too, so people will want to do it... kwim? so i'm more likely to pick a cute carrier like my babyhawk or angelpack or a pretty ring sling.
now, the baby's outfit... i don't think about it too much. after all, you're not going to see much of it with the baby in the carrier. but i tend to have very neutral carriers (blacks, browns, tans, a little light blue/turquoise thrown in) and most of my clothes are pretty neutral too, so it's not really very hard to be matchy-matchy