I moved to Baltimore last year, and I have to say it has been quite an adjustment. But I moved from the cold, cold midwest, so the weather has been a HUGE plus here, not a negative for me at all. I've lived in New England and New York, and the weather in Baltimore is much warmer and more pleasant. My daffodils are coming up in early March -- unheard of in the other places I've lived.
There is crime here, but it's a city. I used to live in Manhattan, and there was lots of crime there, but nothing ever happened to me there (I didn't own a car then). The northern part of Baltimore is the safest and IMHO the nicest, but the other areas mentioned in previous posts are nice too. You have to be street smart here, never leave anything visible in your car, lock your car and house, etc. Don't be stupid, and you are pretty safe. Laptops left on the front seat of a car are likely to be stolen!
It's not very crunchy (though neither am I), and I've had a hard time finding alternative docs, which can be a problem. But they exist, you just have to look harder for them. There are lots of farmers markets, CSAs, natural-foods restaurants, etc. Some crunchiness if you look, but less than I'd like.
People are generally friendly, and yes it's an intellectual place. Lots of doctors and university professors, tons of PhDs roaming around Baltimore doing all sorts of green and groovy things. Large artistic community here, much to my surprise. It's not quaint though. The roads are very bad (ie full of potholes) and the entire city has a slightly forlorn look to it (except for the Inner Harbor, which looks slick). I've never set foot in the really bad neighborhoods, and I don't intend to. It's a segregated city that has not dealt effectively with its long, long history of racism, IMHO.
It is expensive to live here, but probably cheap compared with CA. There are precious few SAHMs, making it harder to live on one income, especially with the so-so schools. We are in Roland Park Elementary school, which is supposed to be the best public elementary in Baltimore. It's OK, but if you have the money, private schools are better.
Living in the city is great, however. Lots of things to do, and Baltimore is small enough that everything is close. We like the museums, the inner harbor, restaurants, shopping, etc., and nothing is more than 20 minutes away when you are in the city. I never feel bored here, as I did in the suburb I moved from. It's a dynamic city that has many challenges, but many opportunities.
Best of luck, whatever you decide. Change is difficult, but it has its rewards.