not really.. depends on your situation but just be aware that whatever problems you guys had before, are probably going to remain.
Hun sometimes a good thing comes from bad ideas ;) go for it if its feels right
If you want looks, then don't date him.
You said he's 4 inches shorter, than if it keeps bugging you, don't date him even if you have a bond or whatever because that little height thing will keep bothering you.
Than forget the height thing and be with him. Looks aren't even that worth it. They fade away with time.