Just take it as a compliment.
I've had it happen many times in my life and it's just a curiosity to some dudes. I wouldn't be offended, if I were you.
You ever been somewhere with those huge Bathtub Urinals. Here your are with a 4 incher when soft and the dude next to you is "Mister Ed" kinda embarrassing.
Well maybe not embarrassing, but you think "Curse My Average Genes" ;-)
If I was male I would take it as both..