gptq

Comparison of different quantization techniques and their impact on inference time. And how to optimize inference time with quantizations using vllm.

Local LLMs-Series: In this part I will show a comparison of different benchmarks with quantized models. The results of official benchmarks differ alot from custom benchmarks.