We're on Year 6 of this now, so it's kind of old hat (although the restrictions have varied depending on which girl was allergic to what - we've had holidays where I and a wee one were dairy/soy/egg/wheat/legume/peanut/treenut/fish/shellfish free, and holidays where the only thing being avoided was egg, and most everything in between. The egg is our constant though.
At this point, my family knows what we can and can't have pretty well. Things that might have the allergens in them are set aside before the allergen is added (ie sweet potatoes or whatever) -- I always bring rolls for everyone, so we can have egg (and sometimes dairy/soy) free rolls, and I always bring a dessert (since we are usually dairy/soy/egg free it seems). With the wide variety of foods available at the meal, I can usually find things that are safe.
We always bring a big laundry basket full of snacks and 'just in case' foods so that if there is a meal which we can't eat, we have something for the girls and me to eat. This used to offend Mom, but I think now she just accepts it as a weird thing we do.

It's a necessity - especially when my diet was really restricted. And frankly, often whatever the girls are eating is sampled by their cousins etc. so we need to have plenty.
When we are eating with the ILs here in town for a holiday meal, I am more aggressive. I bring about half the food, and I monitor how things are cooked more closely. They aren't as conscious of food allergies, and aren't as careful, so if I don't watch closely and make sure we've got a good selection of safe food there, then things get really awkward while the dds and I don't eat much and everyone else chows down.

The thing that makes me nervous is little kids and candies etc. Cross-contamination can happen so easily in those situations! My sisters and their families are very conscious of it, though, and frequently send the kids into the bathroom for handwashing to make sure hands are clean, and always check with me or dh before offering a food to the girls.
One piece of advice for people who are just adjusting to this - if the food is served buffet-style, make sure you're at the front of the line for the food. It doesn't matter where you are, it seems like there are always a few people who will drip fruit salad dressing into the cranberry sauce, or use the spoon from the green bean casserole to dish up some turkey, or whatever.

If you're first in line you won't have to deal with that.