The biggest obstacle of course is dealing with deliverability problems. As most posters have commented here, getting people to accept your mail, and accepting mail from reputable sources isn't an easy problem to solve. At FastMail, a considerable amount of time is spent ensuring customers don't need to worry about deliverability problems or spammy inboxes.

To me, that's the biggest reason not to spend time running your own infrastructure for mail. Getting people to accept your mail reliably, is much harder than adjusting some postfix configuration.