The downside of QBE is that it then requires an assembler and a linker. And QBE's only input and output is still text.
Plus the "frontend -> QBE -> assembler -> binary" process is slower than "frontend -> LLVM -> binary". And LLVM is known for being a fairly slow compiler.