The assumption: a bigger model means better answers

AI vendors sell models in tiers. The largest models score best on general tests and usually cost the most for each request. So when answers about your orders, prices or stock are wrong, upgrading looks like the obvious fix.

But a larger model does not know your business any better than a smaller one. It has never seen your price lists, your customer accounts or the rule that cancelled orders do not count as sales. If those things are missing, a larger model guesses more fluently. It still guesses. Why AI gives wrong answers about your business data explains where those guesses come from.

The evidence: structure closed most of the gap

Buffaly published a test of this in Make a Smaller Model Behave Like a Frontier Model: Put Buffaly Around It (Matt Furnari, Buffaly blog, August 1, 2026). Seven models answered the same 150 questions from MedQA, a public benchmark built on medical licensing exams. Each model answered every question twice: once on its own, and once with Buffaly running a required lookup step first.

The published results:

  • GLM 5.2, an open model, on its own: 101 of 150 correct.
  • The same GLM 5.2 with Buffaly around it: 144 of 150. Same model, no fine-tuning.
  • GPT-5.5, a frontier model, on its own: 146 of 150.

The gap between the open model and the frontier model went from 45 answers to 2. The model did not change. The step around it did.

The result has limits worth knowing. The questions were medical, not commercial. Not every model gained: in the published table, Gemma3 27B scored 117 both times. And the frontier model on its own still scored highest. The point is not that model size never matters. It is that structure moved one model much further than its size did.

Illustration titled Structure closed most of the gap. Three bars compare an open model alone, the same open model with structure, and a frontier model alone. The bar for the open model with structure nearly reaches the frontier model. A bar at the bottom reads: Add structure before you buy size.
Figure 1. In the published test, adding a required lookup step moved an open model close to a frontier model. Bar lengths are illustrative, not to scale.

The cost of reasoning through the same work every time

Model price is only part of the bill. The other part is how much text you send the model on every request. Many AI setups paste records into the prompt and ask the model to work out the same steps from scratch each time: find the customer, find the price list, check the rule, do the sum.

Buffaly's launch article, Introducing Buffaly (Matt Furnari, May 11, 2026), describes two changes to that pattern:

  • The model works with references, not copies. Instead of the whole record being written into the prompt, the model is given a reference to it, and the system's own code does the work on the record.
  • Repeated work becomes a procedure. When the same request comes up again and again, the steps are turned into code that runs, so the model does not reason through them every time. The article reports that in one real task, token cost dropped by 79.7%.

A token is the unit AI providers bill by, roughly a short word or part of one. Fewer tokens per request means a lower bill for the same work, whatever model you use.

Suppose, as an example, a flooring distributor's staff ask an AI assistant dozens of times a day whether an order has shipped. If the assistant reads the whole order history and reasons it out each time, every question costs the same. If checking an order's status is a procedure the system runs, the model only has to recognize the question and call it.

What to spend on instead

If the budget is fixed, spend it in this order:

  1. The data model. Define your customers, products, prices and orders, and which system holds the true value of each. Design the data model first explains how.
  2. Tools. Turn the questions and tasks your team repeats into procedures the AI can call, such as checking an order's status, looking up a customer's price or finding stock across warehouses.
  3. Guards. Put the rules that must never be broken into code: no price below cost, no invoice without approval. Can you trust an AI agent with orders, prices and invoices? explains how those checks work.
  4. The model. Then choose the model. With the first three in place, a smaller or open model often does the job, and you can test that on your own questions before paying for a larger one.

The structure also keeps paying off. AI that learns your business while it works describes how a system can add new procedures as it meets new work.

Illustration titled Spend on the structure first. A stack of three blocks labeled Data model, Tools and Guards supports a smaller block labeled Model on top. A bar at the bottom reads: The model sits on what you build.
Figure 2. Most of the value comes from the data model, tools and guards. The model sits on top of them.

Questions to ask a vendor

  • If we used a smaller or open model, how would your results change on our questions? Can we test that?
  • Does your system send our full records to the model on every request, or references to them?
  • When the same request comes up every day, does the model reason through it again, or does the system run a procedure?
  • Which of our business rules are enforced in code, and which depend on the model following instructions?
  • How much of what we would pay you is the model, and how much is the structure around it?

Send us the question you are paying too much to answer

Tell us one question your team asks an AI every day and what it costs you now, or one it still gets wrong after an upgrade. We will show you which parts belong in the data model, which should be procedures, and what size of model the rest needs. Tell us about your current system or see how we run the work for you on full service.