So you're worried about them calling it sherbert rather than sherbet but you call it ice cream when it isn't? Sounds like your whole family had a problem.
I grew up in the Midwest in the 1950s-60s, and it was always pronounced "shur-burt". I have no idea why.
And a chest-of-drawers was pronounced "chester drawers". I always assumed they were named after some guy named Chester.
Isn't "shur-burt" correct?