I am 5'1, 105 pounds and carry my 20+ pound 16 month old in a Tough Travler...not just around the house and yard, but orienteering...XC skiing, etc.
Tough Traveler has a web site...toughttraveler.com I have to warn you, their gear costs more than other brands, but they also offer a variety of styles to fit most body types, especially those like mine. You can look on their web site to see what type you should use, then either get one of their reconditioned ones (nice, they are less $$ but come with warranty) or try to find one on eBay. They do hold their value well and you can resell easily when you're finished with it.
When I use my backpack, I ALWAYS make sure I have on sturdy footwear, for me these are Vasque Gortex boots with good soles and custom inserts made for my feet. I am blessed to have good carrying feet - size 8-9 on my small frame, means fewer pounds per square inch and more stability and balance. I also wear comfortable clothing such as sports pants and a stretchy shirt. This helps me feel less bulky and lets the backpack conform to my body contours. I've had men tell me that I need to wear the band around my waist, higher up, but I find that adjusting it right over my hips suits me really well.
The Tough Traveler has all sorts of EASY adjustment straps you can use. Even in my ski gear, I can easily make adjustments for going up or down a slope. I use their accessories too...a sun and wind screen, extra carry bags that hang off the side, and another one in back too. It has a 5-point harness which is important if you should happen to tumble or if you have a climber.
My toddler loves going out in his backpack. For me, it's a lot more practical than a sling (but I have one of those too) and I think it's safer when you're cooking or hiking.
Well, Good Luck finding a back pack that fits you and your baby.