In theory Swift's arrays are little more that a c array + length, and this is how we have been encouraged to use them.
The issue here is nothing about "managed code", but about how the Swift compiler behaves without any optimization passes. Currently (and it should be noted that this is a recognized issue), the unoptimized builds are orders of magnitude slower than ObjC because Swift will auto refcount code that in ObjC is either not ref counted at all, or uses manual RC. After optimization, Swift can eliminate many uses and put things on the stack etc, but we are at the mercy of the optimizer for that.
Compare this to the (compararively) predictable behaviour you have in C.
Yes we can in theory use UnsafeMutablePointer everywhere, but in that case it's much more efficient to write the function in C instead.