SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

What happened: Researchers introduced SparseDecoding, a decoding-aware pruning framework for LLM inference that builds calibration matrices from layer-wise activations collected during dense-model autoregressive generation, excluding prefill, and adds an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Key Why it matters: The authors report the method consistently outperforms standard fixed-text calibration on long-form generation benchmarks, addressing the distribution shift between pre-collected natural sequences and self-generated decoding tokens.

Researchers introduced SparseDecoding, a decoding-aware pruning framework for LLM inference that builds calibration matrices from layer-wise activations collected during dense-model autoregressive generation, excluding prefill, and adds an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Key

The authors report the method consistently outperforms standard fixed-text calibration on long-form generation benchmarks, addressing the distribution shift between pre-collected natural sequences and self-generated decoding tokens.