A less hyped inference engine with INT8/FP16 inference supports on both CPU / GPU (cuda).

Model supports list: GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, LLAMA, T5, WHISPER

( Found this library during my research on alternatives to triton/FasterTransformer in Tabby https://github.com/TabbyML/tabby)