I'm feeling like a fat a$$ these days! I wear a large KKAFP, and my husband does too even though he's not fat, just built. And neither one of us have it anywhere near the smallest setting. Course my boobs are gigantic. The pouch definately feels like it fits right though, and I have almost no experience in wearing babies. My 2 yr old didn't like to be worn. We use our KKAFP everyday, at least 1/4 of the day now.
Are you pulling it up and over your shoulder before you put your baby in it? I found this makes it too tight to put the baby in, though once the baby is in it I pull it up and over my shoulder and it fits just right. I read the baby should hang around your belly button.