No, actually, you don't. Let's use a round number: $100,000. Over the course of a typical 30 year loan, calculated at today's rate of 4.5%, you will pay an additional $82,000 on your mortgage loan. So your house costs you $182,000. If, however, you have $100,000 in diversified investments, with compounding interest at a fairly reasonable rate of 4% interest over 30 years, you will end up with about $325,000. So, in truth, taking out that $100,000, you end up with $60,000 after taxes and penalty... and it just cost you close to $150,000 PLUS the taxes and penalties. It just does NOT make any sense. Unless you LOSE money over 30 years (which is possible, but not likely if you are smart about how you invest), keeping the mortgage (with today's rates) and continuing with your 401(k) will most assuredly put you ahead.
ETA: You are probably looking at your total payment instead of just P/I. You're including taxes and insurance, and those you have to pay no matter if you own outright or have a mortgage. So, over the course of 30 years, yeah, insurance and taxes are going to double how much you pay, but you have to pay those anyway.
LOL this is getting beyond my mental reach... or maybe it's just 'cause it's bedtime lol. I think I get what you're saying but I will have to re-read it when I'm more awake!!