It is supposed to be snug, otherwise you'll be carrying to low and your back will pay. I know when I first had one, I was like no way was I getting my baby into this. You'll also be losing some of that postpartum weight so as your bab grows, you'll probably getting smaller. Also, as he weighs more, his weight will stretch the pouch out more.
Are you snapping at the largest setting? If you are, you may be able to go up a size as there is some overlap between the sizes. The two loosest row of snaps on a medium are the same as the two tightest row of snaps on a large. I'm sure Kangaroo Korner will be helpful on helping you decide whether to go up a size.