I don't think you have to be miserable, and I'd consider just about anything short of 55 as "early retirement." I mean, student loans aside most people start out at a living wage and get raises from there. Instead of buying a bigger house, why not put that money to work and start growing it?
I don't know if this is an actual rule, but I'm pretty sure if you're invested in a good ETF and reinvesting dividends, you should be able to double your money every 10-11 years, so the more you can pump in on the front end the better.