For lost email I'm inclined to agree with you. There would need to be some mitigation there (encourage or enforce users to have a backup email, etc).

But for stolen email credentials, one of the first things an attacker is going to do is start going through your email archives to see what services you use, resetting all your passwords to important services.