OP, that sounds like a tough situation. It looks like your post might have been missed so I'm bumping it up for attention. Anyone have experiences to share?
I'm a father of a 4 year old nearly 5. My divorce just ended and I have full custody thankfully. My xw is supposedly bipolar but she is a complex case. Maturity issues seem to be part of that diagnosis and the level is stupefying. If he is a felon then my recommendation is the following: assuming you're in a fairly healthy place emotionally, I'd pursue custody. Record conversations, document everything, texts, emails, everything. Assess your exposure and expect nothing from him. Do not discuss anything other than your daughter. Find friends and family who can help you get as much time as possible. Divorce yourself from everything regarding your relationship and understand that at this time it's better for her to be with you all the time. Surround yourself with positive people. Distance yourself from the abuse and do not respond to provocative communications.
Don't get riled about every little thing, it will drive you crazy. Even if it is illegal to record conversations, get him to admit things on the phone and keep a journal. The court is only interested in what you can prove. Use email to get admissions. Be manipulative to get admissions, by that I mean temper the conversations so he is comfortable admitting these things in email. Play it like a game, it will pay off. This requires you to be stoic and smart. It's a chess game.
Let go of any ideas you had of how it could be or what your daughter is missing out on. Place your sights on the best you can make it for her and you on your own. Consult a child psychologist if you think it would help, but find a good one. Talk with her, bond with her, let her know it's ok to feel mad, sad, or anything and that you feel that way too so she is not alone in her feelings. Make your home her home. If she asks why he speaks to her this way, explain that daddy is having a hard time and that his brain is different and that is why he acts the way he does, make it about him so she doesn't blame herself. Get her involved in social activities, playdates at home, be there for her. Allow her room to process and act out and make sure to facilitate her feelings by keeping language simple, concrete, but that it's good to talk about her feelings and cry if need be.
When he says inappropriate things, give her confidence that isn't true and be understanding of the hard time she is having because of all of this. Mirror behavior, this is so important. Set the example because he isn't. You don't want her coming up with her own answers so make sure to ask her questions from time to time. Validate her feelings and frustrations. Use your life experience to share.
My ew did all the same things, even medicated my daughter to get her to sleep early so she didn't have to parent. If he is on medications where he shouldn't be driving and is using pot, use that for your case. Try and get this documented so any time he will eventually have will be supervised visits. Then you can be worry free.
To my astonishment, my xw hasn't seen or spoken to my dd in a month and has no plans to. Is after a year and a half of fighting for custody. She is a major pot head, on tons of medications, and it's only gotten worse. But my dd and I are well. I don't even recognize the person I married and neither does my daughter.
Get family involved and make your lives routine and fluid and full of positive encourage,ent for her.
I'm on the same boat as you. My son's father is making my life a living hell as well. And he's bi-polar, acts like he's 12, lives at home with his mom, and smokes pot like 100X a day.