Judging from what the compiler team at Apple regularly write in forums and on twitter, it seems that they consider the performance issues for arrays (actually, this happens in other circumstances as well) to be a very real issue.
When you say "swap for a C array", then I suppose that you mean we allocate a chunk of memory like in C, then access it using unsafe methods (as there is no "C array" in Swift). Unfortunately, working with UnsafeMutablePointer and friends is extremely verbose and clumsy, which makes it an extremely bad experience.