From Token Maxxing to the AI Pilot Trap
AI pilot programs are experimental projects that let enterprises test generative models on narrow use cases, but they often stall before becoming production systems that deliver clear, measurable business value at scale. The gap between promising demos and proven impact is now visible across large technology firms. Internal “token maxxing” culture encouraged heavy model usage: Meta employees competed on leaderboards, Google highlighted quadrillions of monthly tokens, and similar contests spread elsewhere. The result was soaring AI activity, colorful prototypes, and—eventually—eye‑opening bills. Amazon shut down an internal token leaderboard, and Uber ran through its AI coding budget for 2026 in months, prompting leaders to ask what they were getting in return. These stories reveal a deeper pattern: enterprises can now measure AI usage in obsessive detail, yet they still lack a reliable way to connect that activity to AI business value.
Measurement Overload, Outcome Blindness
Enterprises have never had more data about AI activity. They can track tokens consumed, prompts sent, code generated, and GPU utilization across every AI pilot program. Uber, for example, can see that around 10% of code changes come from autonomous agents, and leaders talk about employees gaining “superpowers” from these tools. Yet President Andrew Macdonald concedes that “it’s very hard to draw a line” between those usage statistics and useful features for riders or drivers. This is the new measurement trap. Counting activity is easier than proving value, much like the old “lines of code” metric that rewarded more code instead of better software. Today’s AI dashboards show busy models and faster task completion, while product quality, margin improvement, and customer satisfaction remain weakly linked, or not linked at all, to the flood of AI tokens and generated artifacts.
Pilot Purgatory: When Demos Never Become Products
This disconnect pushes organizations into what Kore.ai’s Cathal McCarthy calls “addiction to pilots.” Teams spin up conversational agents, coding copilots, and analytics assistants that work well in controlled demos. Usage looks strong, and leaders see a stream of small wins. But the systems often stop at the proof‑of‑concept stage. They are not hardened for governance, security, and integration with core workflows. Domo’s Ben Schein notes that you can prototype a slick AI demo in an afternoon, but “you can’t vibe code governance, security, and distribution.” Without that step, pilots multiply while real enterprise AI production lags behind. According to industry survey data referenced in the discussion, 79% of organizations report individual productivity gains, yet only 29% see significant ROI and 95% of generative AI pilots show no measurable impact on the P&L, a stark illustration of pilot purgatory.
Where AI ROI Measurement Breaks Down
The core AI ROI measurement challenge is not technical; it is organizational. AI accelerates individual execution, but most enterprises are organized around cross‑team workflows, approvals, and dependencies. When one team speeds up, bottlenecks shift to integration layers where work crosses systems or departments. Local optimization can even increase friction if stakeholders cannot absorb the extra throughput. That is why executives like Uber’s Andrew Macdonald say the “link is not there yet” between AI usage metrics and customer‑facing results. Organizations track inputs—tokens, agent actions, code commits—but struggle to attribute outcomes such as higher conversion rates, lower churn, or improved margins to those inputs. In many companies, no shared framework exists to define which AI‑enabled tasks truly create AI business value, how to assign ownership for those metrics, or when to retire pilots that cannot prove their contribution.
Escaping Pilot Mode: Designing for Production Value
Executives from Domo and Kore.ai point toward a way out: design AI initiatives around production value metrics from day one. That means defining a small set of outcome measures—such as time to ship features, resolution rates in customer support, or error rates in financial processing—and instrumenting systems so AI contributions are visible at that level. Pilots should focus less on low‑stakes “wow” demos and more on pathways to governance, security review, and integration into live workflows. McCarthy argues that real organizational learning happens only “at production scale,” where systems meet legacy tools and real customers. Schein’s warning against token scoreboards points in the same direction: treat token consumption and code generation as diagnostic signals, not goals. Enterprises that reframe AI pilot programs as experiments in measurable business change are most likely to turn usage into durable enterprise AI production.