I donated a lot of my clothes last year. I don't know if it will help you, but this is what I did:
Most of what I got rid of were my 'skinny' clothes. While I lost a lot of weight after I had dd, I gained some back after about a year. I don't know if I'll lose the weight again or not, but even if I do, my body has changed such that even if I were the same weight, the clothes wouldn't fit well. It was hard for me to admit that because I had really nice clothes, the kind we can't afford to buy anymore. I did keep a few pieces that seemed like they could potentially fit, but only if they were really special and if they could be altered if need be.
I had very little that wasn't nursing friendly, and most of it fell into the skinny category, so that made it easy. But, honestly, is it worthwhile to keep clothes you may not be able to wear for years? Will they still be stylish, or if that's not an issue, are you sure you'll still like them? My dd is 3 and just weaned herself. We're thinking of having another, so in a soon as year's time, I could start nursing again for a few years...For me it doesn't make sense to keep things I will potentially get so little use out of for the next few years.
I've kept all of my maternity clothes. I don't have that much, though. I have four shirts, 2 skirts, 1 dress, and six pairs of pants, two of which are jeans. I found that to be enough, but I didn't put on that much weight, and I didn't really start showing until well into the sixth month, so I was able to make do with my looser-fitting regular clothes for quite a while. I know that not everyone is so lucky.
I really hear you on wanting to hold onto things because it's hard to find things you like and fit. I live in a country where I am considered plus size (I'm now a size 10 in the U.S.), which means there is little available that isn't dowdy, and almost everything else is cut to flatter the equivalent of a U.S. size 4. So clothes for larger sizes are just larger versions of that stuff. Add in the fact that I don't even like the style of most of the clothes...yeah, getting rid of the beautiful clothes I had was sooooo hard, but I had to admit to myself that they weren't as useful to me as I needed them to be.
So that's just my experience. I don't know if you'll find it helpful. If I were you I would just think about what kind of clothes you need for the way you're going to live for next several years and how well the clothes you have serve the purpose required of them.