Well, my hubby and I bought this 3 (conforming) bedroom house when our oldest was a month old, and we were planning on having just two children. Eight years and four children later, yeah. They're just going to have to share rooms, and have gotten to the point that they almost flip out if they're not sharing a room with someone (hey, being lonely isn't always fun!). Even when Grandma or Auntie visits, they get to co-sleep with whomever in the 'guest' room that the 19mo and 5yo boys sleep in (the 7yo boy and 3yo girl are in the bunk beds in the 'boy' room, so it's pretty much just all a house of musical beds. The kids think it's great fun sharing a bed with Grandma or cuddling with Auntie when we have a full house of visitors though, it's cute. I'm sure having more space as they get bigger would be nice, but our house is still fairly decent sized at 2100sf.
Here's my caveat though. We homeschool (as in, I don't/can't send them off to school every weekday to get that few hour break to breathe or dig out the house from mess). Hubby works from home 80% of the time and needs a dedicated dad-only door-locking office. I do a boatload of canning and preserving and food stockpiling to last during rough times or just the winter, a ton of cooking/prepping, hubby has a nice firewood stash in several places around the yard and so on (we heat solely with wood). So while I dream of a better/slightly bigger kitchen and a true, in-ground dirt-walled root cellar, this is our house and we're *not* moving unless we can pay cash or something similar. We were unemployed for several months when I was pregnant with the baby, and our expenses, including our mortgage and utility bills, were just under what he got from unemployment. Which so, so, so many people have been shocked by since unemployment for them wouldn't even cover their house loan, forget the insurance and PMI and taxes. Even friends here that know how cheap I am and know we work(ed) hard to pare down our bills in the first place.
So anyway. Do you really, truly need a bigger house? Or just think you do? For the first several years I really didn't like this house - I was always dreaming of the next, bigger, better one. Until I just somehow fell into the 'bloom where you're planted' mindset. And now, my house feels like the [almost] perfect size. Sure, we wouldn't mind more elbow space acreage-wise and the main bathroom has zero floor space and you can't close the door if someone's on the toilet or standing there and it's orange, but it's ours and it's quirky [and affordable!]. ;) If we get around to minor changes at some point, groovy. If we don't, that's fine too. Plus I always think back to my grandmother... she raised 4 boys in a modified chicken coop. There was the living room with sofa (where one uncle slept), two bedrooms (one for grandparents, one with a triple bunk), a kitchen, and outhouse off to the side. That's it. Their fridge was the creek, half their food lived in/came from the barn. If she could do that, even with her MIL living across the gravel drive, alone, in the 2-story-plus-basement farmhouse, I have no reason to complain, you know?
My opinion may change in a few years when my 7yo turns into a teenager and towers over me, but, well... I think I'd rather have a mortgage burning party at that point in time.