SignalTech/AI
Not Everything Needs an LLM
High-priority alert based on source significance and keyword relevance.
Context
What happened: PostHog published findings on using decision models, which return typed answers with probability scores, instead of LLMs for simpler tasks. Several use cases have reached production, including gating expensive LLM calls and routing requests.
Key numbers: Decision models are up to 444.6x cheaper and 193.6x faster than frontier
Why it matters: The post argues decision models make uncertainty explicit, since a calibrated 0.8 score is right about 80% of the time. It also warns thresholds do not carry over between tasks and models, so they must be set per task and per model.
What changed
PostHog published findings on using decision models, which return typed answers with probability scores, instead of LLMs for simpler tasks. Several use cases have reached production, including gating expensive LLM calls and routing requests.
Why it matters
The post argues decision models make uncertainty explicit, since a calibrated 0.8 score is right about 80% of the time. It also warns thresholds do not carry over between tasks and models, so they must be set per task and per model.