It Still Hurts

So I ended up having an affair with a neighbor that started last year and ended yesterday. Both of us are married and it was just supposed to be about sex, not other attachments. I got attached and I do believe he did too. The big problem is that we all became friends, him and my husband and me and his wife. Our kids are inseperable. They had another couple that were best friends, we will call them H2 and W2 that whenever they were over no one else was allowed to come over, or if we could we were always made to seem like an outsider. On top of that W2 and my affair guy was always touchy feeling, they had a very close relationship. So close that it caused problems at their church and around friends. Yet I don't think they were sleeping together. Than again maybe I was wrong. At the beginning of the year my husband and I got into a big fight and I told him what I had been doing. We agreed it would never happen again and we moved on to fix our relationship. At the same time I had called the affair off. Problem is neither me or the guy could be around each other without wanting each other, so it started up again. Fast forward to yesterday when I started deleting people from my online social network that I don't talk to. I had been thinking about deleting W2 (the bestie) for a long time but hadn't done it. She is the type that thinks the world revolves around her and actually believes people are out to take her friends. But the main reason is because we don't talk, either on the network or in person. We will see each other out in the yard and simply say hey and that's it. Like I said when I did try to approach her and ask her to do things with me it was always no. I don't want people on my network that I don't talk to. Anyways, I knew the potential backlash but I deleted her anyway, or at least sent her an email to explain what I was doing and she deleted me before I could get a chance. Immediately my guy deleted me and his wife just deleted me this morning. I talked to him yesterday and was quite upset with him as he was me. I told him that my husband knew about the previous times we had been together and he said well it would be best for it to be over between us. I'm not really hurt over that, I'm hurt that they feel they can't be friends with me because I'm not friends with W2. I guess the whole thing is I have friends that can't stand each other and they have never made me choose. I know true friends won't. I'd like to get the friendship back but I don't see how. They said we can be neighborly and say hi but that's it. I guess in some way I'm sorry that the affair has ended too, but I knew it had to.