The problem is that mail-clients speak IMAP - if you present your own new API you'll need to wrap it in an IMAP-presentation layer before anything can use it.

Sure the simple case of reading/polling/creating mails is not difficult, so your API idea seems attractive initially. But when you say setting up IMAP is hard I suspect you're getting in over your head - IMAP is not hard, even at scale, certainly not compared to writing a damn IMAP-server (I've done it.)

(PS. Besides your API will need more granularity than "GET /user/foo/mailbox" because my mailbox is very very large.)