StockDuty ← Dashboard Nov 13, 2026 01:07 PM ET
SignalTech/AI

Llama.cpp: 6.5x faster prompt processing on Arm with a NEON kernel

Llama.cpp achieves 6.5x faster prompt processing on Arm via a NEON kernel.

What happened: A patch adds a missing Arm NEON (SDOT) inner kernel to llama.cpp's tiled K-quant matmul, which previously had kernels only for AVX-512 VNNI, AVX2 and AVX. On Arm, formats like Q3_K, Q2_K and the IQ family had fallen back to a path that unpacked weights for every dot product. Key numbers: Prompt processing on Qwen2.5 1.5B, 4 threads, 512-token Why it matters: Only prompt processing improves; generation speed is unchanged. The Arm figures remain well below the x86 reference, and the author notes the gain may shrink or reverse above about 16 threads, which was not measured.
A patch adds a missing Arm NEON (SDOT) inner kernel to llama.cpp's tiled K-quant matmul, which previously had kernels only for AVX-512 VNNI, AVX2 and AVX. On Arm, formats like Q3_K, Q2_K and the IQ family had fallen back to a path that unpacked weights for every dot product.
Only prompt processing improves; generation speed is unchanged. The Arm figures remain well below the x86 reference, and the author notes the gain may shrink or reverse above about 16 threads, which was not measured.
Sources