Broken Once, Tested Forever: How Orivox Turns Live Failures Into Replay Tests
By Zahi Raifer
Zahi Raifer is a software developer with 23 years of experience in AI development, Python, .NET and databases, and the founder of Orivox. His background is product development, not sales.
Disclosure: this article was drafted with the help of an AI assistant and reviewed by the author. All facts come from Orivox's own description of its system.
Any system that lets a language model change live customer websites will, sooner or later, do something nobody expected. A request gets read the wrong way. A page has a shape the rules never anticipated. A service on one side of an HTTP call changes and the other side quietly disagrees.
The question is not whether that happens. It is what the system remembers afterwards. In Orivox the answer is a habit we hold ourselves to: every live failure becomes a replay test, and once a behavior is pinned by a test, a later change cannot quietly take it away. This note is about why that habit is practical for an AI editor in particular, and where the tests sit.
Why an AI editor can be replayed at all
Replaying a failure only works if the thing that failed can be captured and run again. A model that writes whole files is hard to replay meaningfully: the output is large, every run differs, and "what went wrong" is buried somewhere in thousands of lines.
Orivox is built the other way around. The planner model (Gemini 3 Flash in our current architecture) never writes files. It answers with a short list of operations drawn from a fixed grammar of 14 verbs, such as set_text, remove, add_class or move_to_page, and every operation addresses one element by its data-ai-id, for example [data-ai-id='a0f0']. Across 24 benchmark runs the planner's output had a median of 141 tokens. Python applies the list with BeautifulSoup, runs its checks, saves the page, commits the change to the project's Git repository and records the turn.
That shape is what makes failures reproducible. An operation list can be validated, logged and replayed before anything touches disk. The input side is predictable too: the planner's context, about 42,000 tokens, is assembled the same way on every turn, from the same rules, the same tool definitions, the page and the recent conversation. When something goes wrong, there is a small, concrete artifact to hold on to, rather than a vague memory that "the AI did something odd".
What gets pinned
The most important behaviors in the editor are the deterministic checks that run on every save. They are written in Python, they do not depend on the model behaving well, and each one is pinned by a replay test so it cannot be lost in a later change.
They are worth listing, because each of them describes a failure we never want a customer to see:
A save may never empty a live page.
Splitting or moving a paragraph keeps every sentence.
A whole-page rewrite keeps every section the owner did not name.
Image addresses are re-checked on every save, so every image keeps loading at the size the page asked for.
The owner's real logo is never swapped for a placeholder.
The legally required privacy notice stays next to the contact form.
The assistant never reports a change that did not happen.
Edits to another page are based on that page as it is now: a page is read before it is written.
None of these is clever on its own. What matters is that they are permanent. A guard that exists only in a prompt can drift the next time the prompt is edited. A guard that exists only in code can be refactored away by someone who did not know why it was there. A guard with a replay test attached fails loudly the moment it stops holding, so the regression is caught by a test instead of by a customer.
Tests on both sides of the wire
The second place tests live is between services. Orivox is two codebases running as four processes: the web app and its job worker, and the editing service and its worker. They share one Postgres database and one Redis, and they talk to each other only over HTTP.
That HTTP boundary is a contract, and contracts break in a particular way: each side can be correct by its own tests while the two disagree. So the contracts between services are pinned by tests on both sides. Releases follow the same dependency as the code: the editing service goes first, then the web app that calls it. A change to the contract between them is made additive first, released, and only then used.
The same idea extends to the second door into the editor. A remote MCP server lets an owner build and edit a site from their own Claude, and its edit tool takes the same 14 operations the in-house planner emits. The editing service applies a list from Claude exactly as it applies one from Gemini: the same checks, the same Git commit, the same transcript. One grammar means one set of guards, and one set of tests that protects both doors.
Where a failure is caught first
Tests are one layer. They work together with the path new code takes to production.
Code never changes on a production server by hand. It is pushed to the repository, pulled into a development environment that has its own copy of the web app and editing service and its own site files, and verified there. Production runs on two identical servers, one active and one standby; a release goes to the standby first and is checked there before it takes a single customer request. Rolling back is moving traffic back.
And some failures only show up from the outside. Every five minutes a probe walks the AI connector end to end, from discovery and sign-in to a real tool call, and emails an incident report on any failure and on recovery. When that probe finds something, it feeds the same loop: the failure is understood, fixed, and turned into a test.
Why this matters more with a model in the loop
With ordinary software, a bug is usually a line of code. With a model in the loop, a bug is often a combination: a phrasing, a page, a rule. That makes it tempting to treat failures as weather, something that passes. We think the opposite discipline is needed. Precisely because the model's behavior can vary, the code around it must not. Only the server decides whether a turn is saved, every edit turn is its own Git commit, and every rule that keeps a site safe is enforced by code and guarded by a test.
The result is a system whose safety only accumulates. Each incident leaves behind a check that holds on a busy day as well as a quiet one.
A takeaway for builders
If you are building an LLM tool that changes real documents, ask two questions early. First: is the model's output small and structured enough to be captured and replayed? If it writes whole files, the answer is probably no, and debugging will stay guesswork. Second: when something goes wrong in production, where does the lesson go? If the answer is "into the prompt" or "into someone's memory", it will be lost. Put it in a test, and let the test hold the line.
Orivox is a free AI website builder for Hebrew- and English-speaking small businesses. Try it at https://www.orivox.pro/.