You need to take care of yourself and your child. He can grow up or go away, let him choose for himself. Meanwhile you do whatever it takes to keep yourself and your child safe and secure. Sounds to me like his accusations are just a way of controlling you through your guilty feelings and insecurities. Immature guys use tricks like that when they want to leave you but want it to be "your fault" so they don't have to feel guilty about leaving.
You need proof? He'll call before long to rant at you on the phone, hoping you'll make another tearful confession, thereby justifying his behavior. Don't buy it. When he calls, if he starts in on you, tell him "Hold it, I've done nothing wrong and I don't have time for this BS. One of us has to do the right thing and take responsibility for our child and, given your behavior, it looks like that's got to be me. Call me back when you grow up." Then hang up on him.
If you show him that you're no longer emotionally dependent on his approval then he'll either man-up and come home or wash out on you altogether. Either way, he'll show is true colors and that'll be a good thing for you.
I will read your story. Please read and comment on mine also. It's called "So Confused".
I posted on your story. Go read what I said. Good luck!