Most people assume money is the biggest obstacle for millennials, but there's another burden that may be the biggest dealbreaker.