Seems to me that he is not being very nice to excuse your inquiry. He should be kind, and answer you politely. Maybe you need time away from him and maybe he needs to start asking you if YOU like being with him?<br />
Try leaving him alone. If he stays away then he is not worthy of your time. If he truly likes you, then he will make sure he has your time.
He is right.