Ask your DH how he would feel if he was the baby, put into a room full of strange giants and just passed around from one to another without any thought for how he is feeling? Honestly! It rankles you because it is not a nice thing to do to another person! Who want's to be treated that way?
Once, an elderly lady came up to my 3 year old DD in the farmers market and started stroking her arm saying, "You are such a nice girl. What a nice girl you are." My DD stared at her, wide eyed. I said to the lady, "Excuse me! Would you like it if I came up to you and did that?" She backed away with a look like, "Well, I never!" When she was gone, DD said to me, "Thanks, mommy, that lady scared me." Just because they're not crying doesn't mean they're not scared. You know the difference, too, as you've explained. Sometimes people are very respectful and are genuinely interested in holding the baby if it's OK, and that's fine! When my first was a baby, she loved being held by everyone, she'd be all smiles if someone asked to take her. But I would never let a stranger come up an touch her anymore than I would let a stranger do that to me.
I agree with everything you've said here, holothuroidea.
OP, I think it comes down to respecting your DD's body and senses. I, too, would ask your DH to put himself in your DD's shoes. It would feel weird to be passed from person to person and I imagine, extremely confusing. Babies are taking in so much visual and auditory stimuli at a party full of strangers.
My son would have freaked out at that age. I recall one time at a family party when my son was on the floor playing (~7mo) and all of my in-laws gathered around him (~ 12 adults standing in a circle around him, all staring at him.). He started wailing and he hardly ever cried, so I knew he was terrified. No one touched him, but there was so much attention focused on him that it overwhelmed him. He's a very social boy now at 18 mo, but it's interesting - he's very affectionate with other kids and babies, but he still doesn't like many adults to handle him. He engages adults, especially our extended family, but is not at all cuddly with anyone but DH and I. I don't consider this a problem - he needs to feel safe. And it certainly makes me feel safe to know that he won't walk off with just anyone.