In the debug trace, it appeared that the Swift hash map got bridged over and over again every time it was to be read. Since this was a set of attributes for CoreGraphics graphics rendering, the dict/array conversion ended up costing magnitudes more than what the rendering itself did.
From what I remember, Lattner (might have been someone else on the compiler team) said that such conversion should only have needed to happen once, and not at every access to an element. Something like that.
This issue started to occur when it was no longer possible to directly create NSDictionary/NSArray from Swift.
I may also misremember things. Best would be to ask on the Apple developer forums.