I would say be direct but understanding, and sugar coat it a little, dont just say "I dont want you back" be gentle.
There is no way to say it so the other party isn't hurt, but remind them that there is a reason this is an EX relationship. Things are no better now, than when you broke the engagement and wish her/him luck in the future. Then walk away. Don't prolong it. What's the point?
no good way other than to tell then the truth