Direct Quote From A Refuser...This is one of the answers to the question "Why do men ********** instead of having sex with their wife"... Link is in my comment below.
"Holy crap! When I first read your question, the first thing that came to mind was "it's impossible!" But then I read your extra info. My marriage was sexless too. I was married for 9 years and we divorced over a lack of sex. It was I who turned to self gratification rather than have sex with her.
She was a large woman but that wasn't what turned me off about her. I was always the one to initiate sex and it was always the same routine over and over. It got to the point where we had sex once per month. Our sex life began going downhill after our son was born but we divorced when he was 4 years old. The stresses of child care was a strain on us. She took a part-time night job. I'd come home around 6:00 pm and she'd be leaving a half hour later. She didn't come home until 4-5 hours later. Meanwhile, I was home all horny and ready to cuddle and make love but with nobody to do it with. I'd wait until my son was asleep before going on the Internet to find naked women videos and pictures to whack-off to. When she'd come home, she'd hint that she's going to bed, but I resented her not being there when I needed her, so I stayed on the computer until she was alsleep and I was done satisfying myself. Once per month I'd go upstairs to do her.
The thing that I hated most about it is that I'm a great lover. I make love with the sole purpose of bringing pleasure to my partner. I don't care about my own pleasure. I resist coming until she's had enough of them. But for me to want to make love to her, she needs to be there many hours before bedtime because I make love for 4 or more hours at a time. Midnight is too late to start sex when you have to be up early every morning.
I'm not saying that he has that same problem, but he has a problem of some sort. He might resent you for some reason. If sex was great years ago but has gotten worse with time, that's a sign. Talk to him frankly about it and see what he complains about. Maybe you can work together to eliminate what he's angry about."