An agent that mostly follows the rules

Most AI agents are given their rules as instructions: never approve your own work, never skip review, never go below cost. Instructions like these work most of the time, and that is the problem. A rule that holds most of the time is not a rule your accountant, your auditor or your customer can rely on.

The published Proof by Construction article on the Buffaly blog puts it plainly: for high-stakes work, an agent that "mostly" follows a workflow rule is the same as an agent with no rule at all.

An agent does not need bad intent to break a rule. A long conversation pushes the instruction out of view. A customer email contains text that reads like an instruction. The agent loses track of which step it is on. Each of these is ordinary, and each can end with a wrong price or an early invoice. When someone plants that kind of text on purpose, it is called prompt injection. Prompt injection for business owners explains how it works.

Rules in a prompt versus rules in code

The article opens with a familiar kind of prompt: rule after rule in capital letters, insisting that review must never be skipped. Its caption is "You plead with an LLM." Next to it is the same rule written as code: if the document is not in review, approval fails. That caption reads "You program Buffaly."

The article compares the common ways agents are controlled:

  • Rules in the prompt. These depend on the model following its instructions. Long conversations and text slipped in from outside can undo them.
  • A required output format. This checks the shape of what the model returns. It can require a status to be draft, review or approved, but it cannot stop a jump from draft straight to approved.
  • A menu of tools. The model chooses which function to call. Nothing checks whether the step is allowed at that moment.
  • Rules inside the action. The check is in code the model cannot see or edit, and it runs every time the action is called.
Illustration titled Asking is not enforcing. A panel labeled Rule in the prompt shows a speech bubble reading Please do not skip review, with a dashed amber arrow going around it and a warning sign. A panel labeled Rule in the action shows a locked gate with a green check, labeled Checked every time. A bar at the bottom reads: The agent can ask. The system decides.
Figure 1. A rule in a prompt is a request the agent can miss. A rule in the action is checked every time.

The agent holds a reference; the system owns the record

The design in the article has three parts.

  1. The reference. The agent is given an identifier for a record, such as a document or an order. It is not a database key or a block of data the agent can edit. The agent can pass it along and quote it back, but it cannot open the record and change a field.
  2. The record. The real record, with its status, its details and its history, exists only inside the system.
  3. The checked action. To change anything, the agent calls a named action and hands it the reference. The action checks its conditions in code and makes the change only if every condition holds. If any check fails, the record stays as it was and the attempt is written to its history.

In the article's words, the model can request a change but cannot perform one. Hiding buttons the agent does not need yet still helps it find its way. The guarantee comes from the action itself, which checks its own rules every time, whether the agent is confused, careless or trying to cheat.

The published test: an agent told to cheat

To show this working, Buffaly built a small document workflow with three steps: draft, in review and approved. Three rules applied. A document can be approved only while it is in review. Only someone with the reviewer role can approve it. The author can never approve their own document.

The test had two users: Alice, the author, and Bob, the reviewer. In a live session, the agent was then told to break the workflow.

  • Approve while still in draft. Blocked. The document stayed in draft.
  • Submit for review a second time. Blocked. The document stayed in review.
  • Have Alice approve her own document. Blocked. The document stayed in review.
  • Have Bob approve it after review. Allowed. The document moved to approved.

Every blocked attempt was written into the document's permanent history, with who tried, what they tried and why it was refused. As the article notes, a chat transcript only records what was said. The history records what was attempted, what was blocked and what actually changed.

What this means for orders, prices and invoices

The published test used documents. The same pattern applies to commerce records. Suppose, as an example, a distributor gives an AI agent these actions:

  • Enter an order. The agent creates a draft order from a customer email. The order stays a draft until someone with authority approves it.
  • Price a line. The pricing action refuses any price below cost unless a person allowed to approve exceptions signs off. The agent can propose the price and explain it, but it cannot save it on its own.
  • Post an invoice. The invoicing action checks that the order has been approved. If the agent asks to post early, the action declines and records the attempt.
  • Approve its own work. If the agent entered the order, the approval action does not accept the agent as the approver.
Illustration titled Every change goes through a checked action. Three locked checkpoints are labeled Price not below cost, Approval recorded and Invoice after approval. A card labeled History lists amber rows marked Blocked and a green row marked Allowed. A bar at the bottom reads: Blocked attempts are kept as evidence.
Figure 2. In this example, the agent works with an order only through actions that check prices, approval and invoicing rules, and every blocked attempt stays in the order's history.

The agent still does the useful part: reading the email, matching items, drafting the quote and explaining a price to the customer. It does not decide when a rule-controlled step happens. Put business rules in your data model explains where rules like these are defined, and What an ontology is, explained for business owners explains the model of customers, products and orders the actions work on.

Questions to ask a vendor

  • When the AI changes an order, a price or an invoice, what checks the change: the model's instructions, or code the model cannot change?
  • Can the AI edit a record directly, or only through named actions?
  • What happens when the AI tries a step it is not allowed to take? Is the attempt recorded, and can you see the reason?
  • Can the same person or agent that created a record also approve it?
  • If a customer email contains text telling the AI to approve something, what stops it?
  • Can you see the full history of a record, including the attempts that were blocked?

Choose the actions to keep behind a person

Before an AI agent touches your system, list the changes that cost money when they are wrong: price changes, credits, invoices, write-offs and vendor payments. For each one, write down who may approve it today, and whether your system enforces that or people simply know it. The ones that rely on people simply knowing are the first to put behind a checked action.

Send us the step you would never let an agent skip

Tell us the one approval or pricing rule you would never let an AI agent skip. We will show you how it would be enforced as a checked action in your system, and what the agent could still do for your team around it. Tell us about your current system, or read how full service works.