I took Effexor while pg with dd1 & am now pg with dd2. I'm concerned, too, but it would be far worse for me to be depressed than on the drugs IMO. Dd1 is a totally normal kid, fwiw, bright & cheerful, all her fingers & toes. I suppose there might be fall out when she hits puberty, but that's an unfortunate risk I felt was necessary.
I hate the withdrawal from this drug, I find that if I miss a dose I can feel it with in a few hours. I think you are dropping your dose too fast, though - you shouldn't be doing this without your dr's help