You just don't want anyone else to have him.
But that's normal, it's the whole possessiveness that monogamy brings with it.
I don't know the full story, but if you refused for three weeks... do him a favor and keep refusing. Because I don't think you really like him, so don't be cruel to him or yourself.
I want you to want me, even if I don't want you basically.
It's an ego thing, very common.