I think confidence is the key to a DH (or anyone else) succeeding at caring for a baby, and until your DH develops his, it's probably best that you not leave the baby with him for long periods of time.
My DH is the sort who's bursting with confidence anyway, and I had a terrible cold and was sore after giving birth, so he started right in caring for DD from the start, which has worked out great.
I guess I started leaving DD with him for an hour or two every once in a while (like once a month if I am lucky) when she was about 6 months old. I waited until she could go for longer between breastfeeding (no bottles used here). Then I bf her and make sure she's happy before I leave, and call to check every, oh, 3 minutes or so!
If you don't want to leave DD, then don't! Screw what everyone else thinks. People think AP is strange. I think wanting to leave your child all the time is strange.
If you *DO* need some time to yourself, or just have things it's easier to get done without DD (like 42 errands that need doing but DH wouldn't be able to figure out how to do), start encouraging DH to care for the baby while you are there and praise him when he does it right, and help when he doesn't. Eventually the confidence will build, the baby wll get used to him changing the dipes, and as the baby grows he'll feel less like he's going to hurt or break something!