Stick Or Twist

me and the wife have been together 20 plus years but many have been hard and full of problems too many to list but we have good sex life when it happens when we are together and happy its good but when we aint its horrific now ive fallen for someone at work whoes the opposite quiet shy and sex might not be instant but i know theres something there should i move on or sick with what i got