Tests as Legal Requirements: Enforcing Determinism in the AI Era

We were taught that a test exists to find a bug. You write an assertion, it goes red, you fix the code, it goes green, and the defect is caught before a user meets it. That is a fine reason to write tests, and it remains true. But it is no longer the most important reason. In a world where software carries legal obligations and where the code is increasingly written by a model rather than a person, the test suite has a second and heavier job: it enforces the boundaries the law draws. The retention window that must expire the record. The lawful-basis check that must run before the processing. The human-oversight gate that must sit in front of the automated decision. The audit-log write that must happen and must be provable afterwards. These are not features. They are constraints, and a test is the only place in the codebase where you can state a constraint and make the machine refuse to cross it.

The boundary is a line, not a behaviour

There is a difference between a test that describes what the code does and a test that asserts what the code must never be allowed to do. The first kind follows the implementation. If the code changes, the test changes with it, and the model that rewrites the function will happily rewrite the test to match. That is the wrong shape for a legal boundary. A boundary is fixed by something outside your codebase: a regulation, a contract, a supervisory authority's expectation. It does not move because the implementation moved. So you write the test to encode the obligation itself, not the current behaviour. The personal data older than the retention window must not survive the job. The decision with legal effect must not reach the user without a human in the loop. Write it as an invariant, and it stops being documentation of what happened and becomes a contract about what may ship.

CI is where the line is drawn

An invariant that lives only in a document is a wish. The place where a boundary becomes real is the pipeline. A failing compliance test has to block the merge exactly the way a failing unit test does, with the same finality and the same lack of negotiation. No override, no "we will fix it next sprint", no green build with a known breach sitting inside it. This is the whole point of putting the obligation in code rather than in a policy PDF: the enforcement is mechanical and it happens before anything reaches production. Software shows its worth when things go wrong, and the moment a change would cross a legal line is precisely when you want the system to stop, loudly, on its own, rather than trust that someone read the policy.

Regulated behaviour has to be deterministic

"Usually works" is acceptable for a recommendation. It is not acceptable for a lawful-basis check or a consent gate. Regulated behaviour must be deterministic and provable: given these inputs, the boundary holds, every time, and you can demonstrate afterwards that it held. That rules out flaky tests, it rules out probabilistic assertions, and it rules out any part of the boundary whose outcome you cannot reproduce on demand. Determinism here is not an engineering nicety. It is what lets you stand behind the guarantee when an auditor, a regulator, or a court asks you to. Auditability over features: the record that the check ran, and its result, is worth more than any capability you shipped that quarter.

The suite is the last human contract

When code is generated at machine speed, the human stops writing most of the lines. What the human still writes, and must keep writing, is the boundary. The test suite becomes the last human-authored contract about what is allowed to ship. The model can generate at full speed inside that contract, and that is fine, because the contract is what stops it generating its way past a legal line. The tests the model wrote to describe what its code happened to do are worth little here; they ratify the output. The tests a human wrote to encode what the system must never do are the thing that holds. That distinction is the whole discipline.

Where tracegate fits

This pairs with the compliance-as-code libraries I have been building. In spring-gdpr and spring-aiact the annotation declares the obligation at the point in the code where it lives: this field is personal data with a retention window, this decision needs human oversight, this processing needs a lawful basis. The annotation states the obligation. The test proves it holds. CI enforces it. tracegate is the tooling I am building around that last part, the direction being to make the boundary visible and enforced in the pipeline as a first-class artefact rather than something you reconstruct from logs after an incident. It is early, and I would rather describe the idea than oversell an API. The idea is enough: treat the legal line as a test, make the pipeline refuse to cross it, and keep the proof.

I ship AI into production. I do it with a human who owns the boundaries, because the boundaries are the part that cannot be delegated to the thing writing the code. The deliverable was never the code. It is the understanding of what must never happen, and the guarantee, mechanical and provable, that it will not.