I don't mind that they ask, as long as they can take no for an answer. I have confidence in most of my friends, but I don't want to pass my baby around when we're in a big crowd. I said as much to one friend, and she said, "Well, why not?"
I was taken aback, and snapped, "Because he's my baby, he's not a doll!" (We were in a huge crowd, it was during services on the high holidays!)
I really don't mind most people I know holding him, and in good and quiet conditions, I let them. He's a social baby though, and these are all people I've seen do well with babies before.
Don't feel bad about protecting your child from people you don't trust. You are the mom. If they say, "It's just because it's your first baby" or "you are overprotective" tell them they are right! They can always flirt with your baby while she in safe in your arms, my son likes that best.