I agree 3 weeks is early! Be gentle with yourself. Also KEGELS! Do them whenever you nurse the babe, or change the diaper, then they'll be part of your daily routine. This will help tone and strenthen you. Drinking teas of any toning herbs may help the suppleness of that area as well- nettles are great. Additionally did you do sitz baths w/ herbs and sea salt pp? This really helped me and my tear healed in less than 2 weeks. Even without a tear, the right combo of herbs can soothe and stregthen- comfrey, calendula, yarrow, witch hazel, and/or lavender work wonders.
My babe is almost 6 months and I still feel nervous around sex. As well as my body is saying (as it should be) no more babies! Breastfeeding keeps me dry and my desire is low. My man and I have done it a few times, beginning at 8 weeks. Go slow, lubricate, massage helps and a glass of wine are all good ways to relax.
I am concerned about one thing- you are drinking coffee and lots of it? Coffee as a diuretic will not help your body stay lubricated as a whole system which will not help your vagina. Also, as I assume you are breastfeeding I hope you are aware that caffeine goes straight in the bloodstream and thus, into your babe via your milk. In moderation caffeine is fine and probably even good for you, but not lots of it! Additionally, at 3 weeks pp your body is still healing, hormones shifting and I would hesitate to be ingesting any stimulant in large amounts- water and herbal teas are best.