We're just getting started also. We have way too much stuff (esp. clothes, it's insane, we still have clothes from before we met....almost 12 years ago)., so we're really trying to downsize. The one thing I can't seem to get rid of is all our books. So first we're going to do clothes, then toys, then odds & ends (knick knacks, bath stuff, etc), then we'll see about the books. I know I should donate some, I just can't seem to bring myself to do it. Would it be so wrong to minimize everything else but keep the books?
ETA: I do feel very lucky, dh completely agrees with me, though he feels the same way about the books so he's no help there LOL).
Go with your plan of going through the other stuff first and you might have a different mindset and are willing to let go of a big chunck of books then. Then again, maybe you will have so more room from getting rid of the other stuff that keeping the books will be easy.