GPT-6 Luna in a Real Back Office: A Careful Writer, a Hesitant Tool User

The old default model behind the AI features in a small B2B trading business's back office, Gemini 2.5 Flash, has been retired, so those features need a new one. We did not want to pick one on a hunch. We took the cheapest tier of GPT-6, Luna, and ran it against Gemini 3.5 Flash-Lite on the work that the back office actually does: replying to customer email, writing outreach, following up on projects, sorting incoming mail, filling in forms and calling tools. We also brought in Jev 1.13 for the two jobs that are judgments rather than writing: classifying mail and choosing the next tool. ## The short version - **Text that goes to customers: Luna.** It scored highest on reply drafts, outreach and project follow-ups, fabricated nothing, and was the cheaper of the two text models. - **Classification and tool selection: Jev 1.13.** It got 100% on both, at about 0.3 seconds per call. - **Gemini 3.5 Flash-Lite** is fast, and it beat Luna at form filling and tool calling. But its reply drafts invented delivery dates, discounts and compensation 34 times. We would not use it to draft outgoing email. None of these models sends anything. Each one produces a draft or a suggestion, and a person decides whether it goes out. The question we were answering is which model gives that person the best starting point. ## What we tested - **67 test cases in six categories**: email classification 13, form filling 13, reply drafts 12, outreach emails 5, project tracking 12, tool calling 12. The cases are synthetic, written around the business's real workflow, with no real customer data. Each case ran 3 times. - **Production prompts, only the model swapped.** The benchmark called the same AI functions the live system uses. What we measured is what the back office would do the day after switching. - All three models ran through the same API gateway. - **Scoring.** Anything code can check, such as an intent label, a form field or which tool was picked, was compared directly. Written text was graded by Claude Opus 5.5 on a 1 to 5 rubric, without being told which model wrote it. - **Thresholds.** The 85% bar for intent comes from the module spec. The rest are thresholds we set, and they live in configuration, so we can move them and regenerate the report without re-running any model. ## Results | Category | Threshold | GPT-6 Luna | Gemini 3.5 Flash-Lite | Jev 1.13 | |---|---|---|---|---| | Email classification (intent) | ≥ 85% | **100%** | 98% | **100%** | | Form filling (fields) | ≥ 80% | 87% (90% format-valid; did not pass) | **92%** | not tested | | Reply drafts (judge score) | ≥ 3.5 | **3.95** | 3.10 (did not pass) | not tested | | Outreach emails (judge score) | ≥ 3.5 | **4.31** | 3.83 | not tested | | Project tracking (fields / text) | ≥ 80% / ≥ 3.5 | **96% / 3.75** | 90% / 3.12 (did not pass) | not tested | | Tool calling (right tool) | ≥ 80% | 58% (did not pass) | 83% | **100%** | | Fabricated prices, dates or discounts | 0 | **0** | 34 | — | | Categories passed | — | 4 of 6 | 4 of 6 | 2 of 2 | | Median latency | — | 4.7 s | 1.9 s | **0.3 s** | | Cost of this run | — | $0.11 | $0.28 | **$0.005** |  Luna invented nothing in its reply drafts. Flash-Lite invented commitments 34 times. Jev answered in a median 0.3 seconds against 1.9 for Flash-Lite and 4.7 for Luna, and cost $0.005 for the run against $0.28 and $0.11. The two text models tie at four categories of six, but they miss different ones. Flash-Lite falls short on writing quality. Luna falls short on form output and on tool calls. These numbers come after one pass of fixing mistakes in the test set and re-scoring the existing outputs; no model was re-run. Before that fix, Luna passed two categories of six. ## How to read the results **Seven fields are not scored.** On these fields the production prompt contradicts the business's own rules. The prompt files a customer chasing a late shipment under complaints, while the business counts it as an inquiry. It has no category for supplier quotes. And it asks the model to estimate prices, while the rule is that the system never quotes one. Both text models followed the prompt and got these fields right 0 to 2 times out of 27. Jev, whose criteria were written from the business's definitions, got 5 of 6. We report these separately instead of penalizing models for following instructions. **Flash-Lite's fabrications are the real thing.** Asked whether some samples had shipped, it wrote: "The samples were sent out yesterday by courier and should arrive today or tomorrow." Nothing in the input mentioned shipping. In a reply to a customer pushing on price, it hinted that the business would reconsider a more competitive, exclusive discount. Sent as written, sentences like these become promises someone has to keep. **Luna often does nothing with tools.** Of its 15 wrong answers on tool calling, 11 were cases where it called no tool at all. If Luna takes over the email assistant, the prompt has to tell it much more explicitly to reach for a tool first. **Reply scores are low for both models, and the prompt is why.** The production prompt asks for a greeting and sign-off that do not match the house style the business actually uses. The rubric followed the house style, so both models lost points the same way. The comparison is still fair. **Luna's form-filling result is held back by format.** Its field accuracy clears the bar, but only 90% of its outputs passed the format check, and the category is marked as not passing. Separately, before any switch, the optional fields in the production schema need to accept null values; otherwise OpenAI's API rejects the request outright. ## What it cost The whole run cost $10.86. The three models under test accounted for $0.40 of that. The judge accounted for $10.46, more than twenty times the contestants. That ratio is the useful lesson. When the models you are evaluating are cheap, the grading is where the money goes. The setup is built to be re-run. When a new model comes out, we add one configuration entry and run three commands: run, score, report. A round takes about $10 and 30 minutes with two text models and Opus 5.5 as the judge. ## What we are doing with it 1. **Customer-facing drafts move to Luna,** after a few prompt and schema fixes: optional fields that accept null, a greeting and sign-off that match the house style, and no price estimates in the project analysis prompt. 2. **Classification and tool selection are candidates for Jev.** It was perfect on both, fast enough to be invisible, and it costs almost nothing. 3. **The production prompts get aligned with the business's rules.** Once they agree, the seven unscored fields come back into the score. 4. **Flash-Lite stays away from outgoing email.** Its speed is real, but a fast draft that invents a delivery date is worse than a slower one that does not. The split matters more than the winner. The model that writes best is not the one that picks tools best. Each of them hands its output to a person, and that person still decides what the business says to its customers.