Benchmarking LLM Compression: FP8, GPTQ, and SmoothQuant with llmcompressor

A Coding Implementation to Compress and Benchmark Instruction-Tuned LLMs with FP8, GPTQ, and SmoothQuant Quantization using llmcompressor

The llmcompressor library facilitates post-training quantization for instruction-tuned models like Qwen2.5-0.5B-Instruct. Using an FP16 baseline, developers can evaluate performance trade-offs across FP8, 4-bit, and 8-bit compression recipes.

Why This Matters

Deploying large language models in FP16 precision often exceeds the memory and budget constraints of production environments. While high-precision models provide the highest accuracy, technical reality necessitates quantization to reduce disk size and increase throughput; however, aggressive compression like 4-bit GPTQ requires careful calibration with datasets like UltraChat to prevent significant perplexity spikes and output degradation.

Key Insights

• FP8 Dynamic Quantization offers a data-free compression strategy for Linear layers while preserving the lm_head in higher precision.
• GPTQ W4A16 reduces weights to 4-bit precision, utilizing 256 calibration samples from the UltraChat 200k dataset to minimize reconstruction error.
• SmoothQuant W8A8 mitigates activation outliers by applying a smoothing strength of 0.8 before 8-bit quantization.
• The oneshot API in llmcompressor provides a unified interface for applying complex quantization recipes to Hugging Face models.
• Benchmarking across metrics like perplexity (PPL) and tokens per second (tok/s) is essential to identify the optimal balance between model size and generation quality.

Working Examples

Implementation of FP8 dynamic quantization using the llmcompressor QuantizationModifier.

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto")
recipe_fp8 = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)
oneshot(model=model, recipe=recipe_fp8)
model.save_pretrained("Qwen2.5-0.5B-FP8-Dynamic", save_compressed=True)

Advanced SmoothQuant and GPTQ W8A8 pipeline for handling activation outliers.

from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.modifiers.quantization import GPTQModifier

recipe_w8a8 = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
]
oneshot(
    model=model,
    dataset=calib_ds,
    recipe=recipe_w8a8,
    max_seq_length=1024,
    num_calibration_samples=256,
)

Practical Applications

• Optimizing edge deployment of Qwen2.5 where memory is limited to 4-bit weight storage using GPTQ. Pitfall: Using generic datasets for calibration can lead to degraded instruction-following capabilities.
• Scaling high-throughput inference services by using FP8 Dynamic quantization to increase tokens per second on T4 GPUs. Pitfall: Neglecting to monitor perplexity (PPL) can result in subtle but critical drops in output quality.

References:

• https://www.marktechpost.com/2026/05/17/a-coding-implementation-to-compress-and-benchmark-instruction-tuned-llms-with-fp8-gptq-and-smoothquant-quantization-using-llmcompressor/