I would recommend interviewing several attorneys - paid and unpaid - to see what the general concensus is about your situation. If most are saying you have a good chance at both sole physical and legal custody then go for it. If most are saying no then you may need to step back.
I found attorneys who only offer paid (or reduced) initial consultations to be much more worthwhile than the free ones. You definitely get what you pay for. I spent nearly $1000 just interviewing, but I feel interviewing a variety of attorneys helped me put my situation into perspective.
You can do customized legal custody - the Ex was open to that. You have final decision making power over education and medical and the burden is on him to take you to court if he ever whole-heartedly disagrees. At least in my state both legal and physical custody can be up for review every year.
Does your state do a conference meeting before anything goes to trial? The case manager at the conference meeting talked A LOT of sense into the Ex
He was literally asking for everything and honestly believing he was going to get it!
Does your state offer conciliation services (sp?). It is where one party requests that you both receive (and possibly the children) multiple mental health evaluations. Then this pyschologist sends their report and recommendations to the judge and this carries a lot of weight. This may be good in your situation since your spouse is unstable yet you have little hard evidence. The hope is that professional will be able to recognize his character and pass this on to the judge.
I had more thoughts, but I'm really frazzled right now... sorry.