IMHO this is a misconception. Dart compiles to JavaScript with decent performance of the generated code. In some cases even faster than comparable hand-written JavaScript (see: http://dartlang.org/performance).
JavaScript engines getting faster also means that Dart code compiled to JavaScript is getting faster. It is a win-win scenario.
Even if other browser vendors won't ship a Dart VM tomorrow the situations is pretty awesome performance wise.
Once Chrome ships a Dart VM a huge user base gets even better and more predictable performance characteristics than would be possible in JavaScript which is incredible but I don't see the browser vendor support/performance angle as a real show-stopper for Dart.
You don't give anything up when you use Dart, you can still call JavaScript libs if you need to (e.g. we do use PubNub.js at https://www.blossom.io and we use it from Dart without any problems).