Think kids .. First priority ... Always
Keep the children away from him, he is not in a position in his life right now to be of any positive effect on the children. The children do not need to see him this way either...when he's on a high, he will not be able to make rational choices or anything when the children are with him and this is dangerous. Talk with him, tell him how you feel and let him know what is going to happen. He must want to clean himself up permanently before seeing the kids and if he wants this bad enough, he will and can do this but until then...personally, I would keep the children safe, not subjected to this type of lifestyle. They are young and need the best environment they can possibly have....one good parent is most important. If they ask you why....you could tell them that dad is not feeling well and is fixing a lot in his life....make it appear his fixing things is like a long job for now until they get older and have a better understanding of why.
It's best to protect your kids from that behavior period. Allowing them to be around a known drug user CAN get YOU charged with neglect in most states! It doesn't matter if he's the father.
Don't reward his bad behavior. Make it known not seeing his kids is a consequence of his actions.