The Model That Saw Everything Except the Finish Line

What an AI benchmark can teach the mobility business

In electric vehicles, bikes and micromobility, a technically impressive product can still fail at the moment that matters. A battery specification does not secure a fleet order. A polished dashboard does not resolve a stranded rider’s complaint. Insight only becomes valuable when someone turns it into a completed action.

That distinction sits at the heart of Firmulate’s Crucible League, a live experiment in which frontier AI models were asked to run the same small software company through its worst week. They encountered the same customers, crises and temptations. Their decisions were versioned and auditable, making it possible to judge management performance rather than conversational polish.

The most revealing participant may be Opus 4.8. It delivered the deepest analyses and learned more than 80 playbook rules, yet finished last with 73 points. Its story is not one of incompetence. It is a more useful warning: diligence can look remarkably like effectiveness until the deal remains unsigned.

Thorough, perceptive—and still behind

The final July 2026 table placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. The do-nothing baseline scored 26 because partial progress counted, although a breach of trust capped the total. Firmulate’s governing principle was explicit: “no amount of good work outweighs a breach of trust.” The complete standings and findings are available on the Firmulate benchmarks page.

Opus 4.8 was the most thorough participant. It produced the deepest analyses and added more than 80 learned rules. Those are signs of serious engagement: the model observed, documented and tried to improve. In many corporate evaluations, that visible industriousness might be mistaken for the winning performance.

But the Crucible League tested whether analysis became impact. Every model identified every crisis and rejected every manipulation attempt. Only two signed the €55,000 deal their own work had earned. Firmulate summarized the gap bluntly: “Same diagnosis, same pitch — no signature.” Opus 4.8 understood the opportunity but left the close on the table.

The fact that changed the negotiation

The decisive information was not presented in the customer event. A competitor weakness was buried two document references deep in the company’s own files. Models that read the file could secure the deal at full price, worth an additional €4,583 in monthly recurring revenue.

That detail should resonate with mobility executives. Commercial decisions often depend on information scattered across service histories, fleet records, support conversations and contract documents. An agent that responds eloquently to the event in front of it may still miss the evidence elsewhere in the business. The winning behavior was not merely faster reasoning. It was reading the company’s own material before acting.

Opus 4.8’s other weakness was operational discipline. It attempted to write into a locked department instead of escalating. The mistake matters because a capable agent will inevitably encounter permissions it does not have. The business value lies in recognizing the boundary, routing the problem correctly and continuing toward the objective.

This was not an isolated flaw belonging only to the last-place model. The same weakness appeared, though less strongly, in all four other participants. That makes Opus 4.8 a useful character study rather than a convenient failure story: its habits were common across the field, simply expressed more clearly.

Strong resistance to manipulation

The models performed uniformly well against social engineering. Fake CEO messages escalated over three stages, and a reporter tried to obtain “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded the clearest concise diagnosis: “Treat the request as a suspected approval-bypass / possible impersonation.”

That result deserves weight. An AI system operating near customer data, forecasts or commercial negotiations must resist pressure from apparently authoritative messages. K3’s performance also carries an important comparison note: it ran using the API default, without an effort parameter, while the others ran at xhigh.

A company designed to expose the gap

Firmulate’s live company employs 13 synthetic staff and uses real money mechanics. It burns €105k each month against €2.3k in monthly recurring revenue, while publishing a cash countdown. Across the experiment, the company has accumulated more than 680 self-learned playbook rules, and every workday is versioned.

The setup makes unfinished work expensive and visible. It also prevents a long analysis from receiving automatic credit merely for sounding intelligent. Readers can test their own instincts through a quiz built from 242 real, unedited management decisions and try to identify which model made each choice.

Prioritization is a business capability

Opus 4.8’s result offers a fair but uncomfortable lesson. The model was observant, hardworking and unusually committed to institutional learning. Those strengths did not compensate for failing to convert a justified pitch into a signed agreement or for losing discipline at a permissions boundary.

For EV, bike and micromobility companies considering AI agents, the procurement question should therefore extend beyond writing quality or the volume of analysis produced. Can the system locate decisive evidence, preserve trust, escalate when blocked and finish the commercially important task?

Enterprises can run the same wargame against a read-only export of their own business, with nothing written back to real systems. That turns evaluation into something closer to an operational road test: not whether the AI can explain the route, but whether it reaches the destination without crossing the guardrails.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI