Low thyroid function (hypothyroidism) could be responsible. If you haven't had your thyroid function tested recently, it would be worth your while to ask your physician to test you.
If your thyroid is normal, you're not anemic, underweight or out of shape, you may just be cold sensitive, perhaps because of the pattern of blood vessels in your skin. (Cold perception comes from skin temperature, but it's your core temperature that's important, and it's probably normal.) Stress can be the culprit here, because it leads to over activity of the sympathetic nervous system, and that constricts blood vessels in the skin, especially in the hands and feet. For this problem, I recommend a course of biofeedback training to learn to warm your hands.
You should see a doctor. Get your blood checked.