I'll chuck DH's stuff, no problem. He's not a terrible packrat, just darn LAZY about some things. I wouldn't throw away anything with emotional value (if I'm not sure I ask), but if I didn't throw away his stuff, he'd have a drawer of boxer shorts full of holes, socks and T-shirts too worn to wear, ditto jeans that are so worn-through they go to the trash and not Goodwill, etc. Sometimes I think that he believes there is a magic fairy in our house that periodically replaces holey clothes with fresh ones.
He doesn't do ANYTHING with paper junk, so I shred the old pay stubs, cable bills, whatever. I don't throw away photos, stuffed animals (of which we have WAY TOO MANY, given that we're both adults!), or other sentimental stuff; I don't go into the shed and suddenly decide that he no longer needs a circular-sanding-thingy.
Seriously, if DH suddenly decided to do the same kind of thing (i.e. has a complete personality change!) and starts cleaning out my panty drawer, the kitchen cabinets, or the filing cabinet, it wouldn't bother me! We each have a solid idea of what the other considers important, so it's really no big deal.