TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving

What happened: TAO is a tool-aware joint scheduling framework for LLM-based agentic serving. It pauses sessions during external tool calls, scores session-node pairs, and uses a dynamic program to place waiting sessions under GPU capacity limits. Key numbers: Up to 26.8% lower latency, up to 36.3% lower mean TTFT, up to 29.5% higher decode throughput versus Why it matters: The authors state existing systems overlook tool-return readiness and rely on heuristics for admission and placement, leading to suboptimal decisions. TAO targets bursty agentic workloads under tight KV cache budgets.

TAO is a tool-aware joint scheduling framework for LLM-based agentic serving. It pauses sessions during external tool calls, scores session-node pairs, and uses a dynamic program to place waiting sessions under GPU capacity limits.

The authors state existing systems overlook tool-return readiness and rely on heuristics for admission and placement, leading to suboptimal decisions. TAO targets bursty agentic workloads under tight KV cache budgets.