When you have to take care of yourself. That is when.
It's great to help others and I do it all the time, but when you are down, then you need to take care of yourself or you are no good to anyone else.
It's not your place to take care of his kids, even though they are family. If he asks you to help that is one thing. But you have to live your life and go one your way. Now I know this will sound rough but I feel I need to say it. No one ever said that anyone had to meet your expectations of being a suitable Mother.
Your brother chose her, and has stayed with her so he has made his decision.
You don't have to fully let go. And if you feel obligated, then you should do what you can for the kids. Otherwise, I think you'll always wonder what would have happened if you at least tried. But you need to take time when you can and separate yourself. Find ways to make yourself stronger. Find your own activities and friends that make you happy. But I wouldn't let go if you are someone those kids need or can turn to in a time of need.