I can't imagine why your BF would have a problem with this.
Perhaps offer to let him sit this one out, if he's uncomfortable, and go without him.
It is certainly not wrong or disrespectful on your part.
Then go without him. If he makes a huge deal out of this reconsider the relationship. You were related to these people before you knew him. Anyone reasonable and emotionally-mature would understand that.
its not like there is bad blood between you and your former sil just talk to your bf hes just feeling insecure about the whole thing I would go if it was me