SignalTech/AI
Sub-1-Bit LLM Compression via Latent Factorization
High-priority alert based on source significance and keyword relevance.
Context
What happened: Researchers released the official implementation of LittleBit (NeurIPS 2025) and LittleBit-2 (ICML 2026), methods that compress large language models into the sub-1-bit regime via latent factorization.
Key numbers: Designed for 1.0 to 0.1 bits per weight.
Why it matters: LittleBit-2 adds an opt-in Joint-ITQ initialization (--use_itq) that aligns SVD-derived latent factors with the binary hypercube before QAT, with no additional inference overhead. The codebase supports OPT, Llama and Llama 2/3, Phi-4, Qwen2.5, QwQ, Gemma 2 and Gemma 3, and Qwen3.
What changed
Researchers released the official implementation of LittleBit (NeurIPS 2025) and LittleBit-2 (ICML 2026), methods that compress large language models into the sub-1-bit regime via latent factorization.
Why it matters
LittleBit-2 adds an opt-in Joint-ITQ initialization (--use_itq) that aligns SVD-derived latent factors with the binary hypercube before QAT, with no additional inference overhead. The codebase supports OPT, Llama and Llama 2/3, Phi-4, Qwen2.5, QwQ, Gemma 2 and Gemma 3, and Qwen3.