I only care about latency from pressing Ctrl+alt+t to typing the first char of the command.

xfce4-terminal does just fine for this. So I just use that.