Like many who have commented here I am a big time Arch linux fan. However if setting up X seems something that you do not want to deal with, you can give Chakra linux a try. Its based on Arch so you get a lean set up. But the installer does some of the heavier lifting for you. Off late they have been moving away from Arch though.

You could also run prelink after the install to make the startup time of your applications snappier.

Often time I do wish there was a more Debian stable'ish equivalent of Arch to put on servers. A distribution that is optimized but does not tip toe the bleeding edge as much and does not have a rolling release cycle.