That's WAY extreme, IMO. But then again, I live in Texas and set my theromostat at 68.
You could get all Dr. Phil on him with a conversation along these lines: Dear husband, I appreciate all that you do to help our money go as far as it possibly can. You are a good provider and we are in good shape. I'd like to discuss something with you that is important to me. Dc and I are very uncomfortable when the house is in the 50s. I'd like to keep the house warmer that we have been. I'd feel much more comfortable if we kept the thermostat at around 60 (or 65) degrees. I'd like us to work out a compromise so that we can all be comfortable in our house.
OR, you can do what I'd do which is say, "I'm turning up the thermostat to 65 (or 60), and unless you want to physically wrestle me away from it, that's where it's staying."
Whichever works best for you.