A. You and your husband have obviously made security a top financial priority. It's laudable that you have so much in savings and more than $600,000 set aside for retirement at your age. But yours is the rarest of situations: You might have too much money stashed in your emergency fund.
To have the eight-month cushion I recommend, you'd need to save $180,000 in the credit union account only if your typical monthly expenses totaled $22,500. Based on the size of your mortgage payment, though, I'm guessing you spend far less. This means that if you intend to keep your current home for the rest of your life, you could indeed take $100,000 from your credit union account to pay off the mortgage and still have sufficient savings.
From a purely financial point of view, paying off the loan makes great sense. It's smart to use some of your savings to retire a debt that costs you almost 6 percent. Savings accounts at most banks and credit unions currently pay a maximum interest rate of just 1 percent; the interest is taxable, so let's just say you're pretty much assured to net next to nothing.
I want to stress that the two of you have made terrific financial choices. If the thought of reducing the amount in your savings account to just $80,000 makes you nervous, then don't do it. You've earned the right to do what feels comfortable regardless of what makes the most money sense.
Next: Buying vs. renting when you're in debt