A lender makes more money when you borrow more money. And most lenders don't really know everything about your finances; they just look at your income and do a quick calculation to tell you what they think you can afford. They don't know anything about how much you're saving for retirement or whether you want to help your kids financially with college.
You need to decide what you can afford to buy for yourself, based on your long-term goals. Always remember: The smaller your mortgage, the more money you'll have for other financial goals.