We have three cats and two dogs, Two of the cats routinely sleep with us and DH is in charge of the litter box.
I haven't worried once about being close to the kitties while pg -- it's only from messing about with their litter leavings that you can catch toxo-p and even then the chances are very low if you've owned/been around cats for a while. It's only a big deal if you're pg and its incredibly rare for someone who has cared for cats previously to catch it. I was told you can build up an immunity to it? I thought it only showed up on a test positive if it were active at the time -- but you can have it, think its a bug and not know it, then it go dormant (test negative), but you can't catch it again after you've had it once?
I could be wrong but I've never bothered testing for it as assumed I got it at some point since I've owned cats for years.
Regardless, my only concern would be if I were the one changing the litter box and if I were doing so sans gloves.
And even if our one cat with claws were to scratch me (he does sometimes by accident) I just go wash it carefully, rinse it with hydrogen peroxide or alcohol and put neosporin on it. I did that before I got pg though as it would always raise a welt on my arm if I didn't, and itch like mad. I do it now as I know it's being super careful about poop contamination but really it's just to avoid the itch.
I'm super paranoid about a lot of things (no eggnog for me, thankyouverymuch) but when it comes to the kitties, so long as I'm not touching the poop, I'm fine.
(And the recomendation I've read is even if you do come in contact with the litter, just wash your hands really well afterward. Same as you would if you were gardening.)
Even if I'm wrong about the test, just avoid the litter box (or wear long gloves if you have to change it), hug your kitty and get a good night's sleep. This is one thing I wouldn't worry about.