I Am In Love With A Coworker

I have been at my job for almost a year. I am recently divorced. (been separated for a long time) my ex has moved on.. However when I started this job, there is a coworker that is a big cut up, and the first time I ever spoke a word to him, I said to myself that this is the person that I am supposed to spend the rest of my life with... We became friends really fast and flirt with each other like crazy and text each other throughout the day.. However he is a single dad with a 10 year old son that he takes care of. he also has a 15 year old from his first marriage. He works ALL the time, and when he is not working he is doing things with his kids. or helping his mom and dad do things. He has not been in a relationship in 7 years. (that is when he got his second divorce) I am also twice divorced. Both of my marraiges should have NEVER happened. I have never in my life been this attracted to someone so fast. I think of him all the time. We did however go on a weekend trip with some friends of his and camped. We had a really good time, but we said we were just going as friends. We did kiss, but nothing else. Everyone at work tells us that we act like a married couple cause we go on and pick at each other all the time in front of them. But when we text he sends me the sweetest texts. I want to spend more time with him outside of work, and I think he does too, but he never asks because he always has "things" going on with his kids. he even told me that he told his kids that he does not have a life because he does EVERYTHING for them. The oldest son, lives with his mother and she is involved with his life, but the youngest that he has custody of, his mother does not do alot with him at all. He came to work with him one day and his son hung out in my office with me for a while, and had a blast. He even asked me if I would go out with his dad. or if I would marry his dad!! lol. he said his dad is lonely... All my friends say that he sounds like he likes me but he is scared. I know that his last marriage his wife cheated on him with a way younger man.. He is 41 and I am 37. One day I kind of got mad at him and sent a nasty text. and told him that I would not call or text him any longer.. he did not text me back for a couple days. then he textd me and when I told him that I would delete his number so that I dont bother him, he said that I dont bother him. Like he does not want me to quit. He is the hardest person to figure out. But when we are in the same room together, and we talk to each other we look right into each others eyes. and when that happens my heart just melts!!!  I can just hear his voice and it makes me smile and tingle. I know this sounds like a high school crush.. But my heart tells me it is much more than that. Should I just give him time to come around and trust that I would never hurt him like he has been hurt before??