It's never really too early to think about performance. All you have to do is try not to act early on that. A lot of architecture ideas we have (sane URLs, caching servers etc) are rooted in performance considerations.