You personally might stay careful, but the whole point of vouching systems is to reduce review effort in aggregate. If they don't change behavior, they add complexity without benefi.. and if they do, that's exactly where supply-chain risk comes from.
I also like the flexibility of a system like this. You don't have to completely refuse contributions from people who aren't whitelisted, but since the general admission queue is much longer and full of slop, it makes sense to give known good actors a shortcut to being given your attention.
I wouldn't do this where it's not clear there was an issue, but for something like the really poor OCaml PR that was floating around, reporting the user to me seems like a logical step to reduce the flood.
I don't know whether that's good or bad for the overall open-source ecosystem.