Tell them the truth, make it clear you aren't expecting all your needs to be catered too. When you're at the meal eat only what has to be okay - don't take chances! Sadly enough, don't trust it when people assure you a dish is alright for you. Most people think "oh, just a little bit won't hurt her", no matter how well you've explained it.
I'm sure nothing I said is new to you, just giving reassurance that what you do is necessary.
Don't eat, if it's poison to your body.
I'd love to say that if you explain, most people would understand, but sadly it isn't true. I've worked for 10 years in the same place and this Christmas, people who know me well gave me little treat gifts and offered cakes and slices that all had gluten in them.
Still you have to be honest and let them know. Your health is more important than anything else.
You could tell them that you are unable to come or when you do go over, practice self control and eat very little of the food that is there.