SignalTech/AI
FDE-Gym: The Last Mile of AI Delivery
FDE-Gym: The Last Mile of AI Delivery
Context
What happened: Researchers introduced FDE-Gym, a benchmark of 95 end-to-end AI delivery tasks drawn from 20 customer scenarios across 18 sectors, where agents must clarify ambiguous requirements and uncover unstated constraints in simulated customer environments.
Key numbers: 95 tasks, 20 scenarios, 18 sectors, 13 model configurations from
Why it matters: The results show a gap between solving well-specified tasks and delivering working systems for real customers. Discovering unstated constraints is the main bottleneck, and weak models often leave the customer worse off than doing nothing.
What changed
Researchers introduced FDE-Gym, a benchmark of 95 end-to-end AI delivery tasks drawn from 20 customer scenarios across 18 sectors, where agents must clarify ambiguous requirements and uncover unstated constraints in simulated customer environments.
Why it matters
The results show a gap between solving well-specified tasks and delivering working systems for real customers. Discovering unstated constraints is the main bottleneck, and weak models often leave the customer worse off than doing nothing.