Harness design

If you don't have a harness, you are the harness.

The best tests do more than check examples. They turn contracts, properties, complexity budgets, and review packets into constraints the system can verify before the reviewer absorbs the variance by hand.

Opinions If you don't have a harness, you are the harness.
Join newsletter

The most expensive test in a software organization is not a slow integration suite. It is a person rereading the same shape of failure every week because the system never learned to check its own constraints. The reviewer becomes the sensor, the control chart, the rollback plan, and the defect classifier. That is not engineering rigor. That is human variance wearing a process badge.

The thesis:

Constraints are system properties. Contract tests, property tests, complexity limits, and review packets are different ways of making those properties executable. The harness should layer those checks, then optimize correctness and quality together. If it cannot inspect them, the human reviewer becomes the harness.

Constraints are not paperwork

A constraint is a statement about what the system must keep true while it changes. "The id is unique" is a constraint. "Every artifact is versioned" is a constraint. "Every agent event links back to the job it served" is a constraint. Once you state those rules, you have more than documentation. You have raw material for a test, tracker, review packet, and production control.

Suppose a job owns many agent actions, artifacts, and decisions. If each artifact id is unique, each revision is versioned, and each event links back to the job, the tracker can use a composite handle like job_id:artifact_id:vNN. That handle is not cosmetic. It lets the review packet answer: which job produced this object, which version is under review, which agent touched it, and which property failed.

01

Contract test

Turns an integration promise into executable examples: this consumer expects this provider message to keep its shape.

02

Property test

Turns an invariant into generated pressure: this rule should remain true across many inputs, orders, ids, and edge cases.

03

Complexity limit

Turns maintainability into a budget: this path cannot keep gaining branches without forcing a design review.

04

Review packet

Turns a human review into a bounded artifact: here are the ids, properties, contracts, deltas, risks, and proof links.

Do not make one test carry the whole loop

The trap is to collapse the process into one giant code check and call that rigor. A single layer can tell you whether one slice passed, but it cannot tell you whether the system is still understandable, reviewable, compatible, recoverable, and worth keeping in this shape. The loop needs layers because each layer measures a different failure mode.

Treat the harness like an objective function with two families of signals. Correctness asks whether the system does what it promised: examples pass, contracts still match, generated properties hold, simulated pressure stays bounded, and live receipts prove the deployed path. Quality asks whether the system should be allowed to keep its current shape: complexity stays bounded, names explain intent, overload behavior is explicit, review packets are complete, and the next operator can resume without folklore.

Layer Correctness signal Quality signal
Example tests Known cases produce the expected output. The case names explain the user-visible behavior being protected.
Contract tests Consumer and provider still agree on the message shape. The contract is small enough to understand and version deliberately.
Property tests Invariants hold across generated inputs, orders, and edge cases. The properties describe real system constraints, not trivia.
Backpressure simulations Queues, retries, timeouts, cancellations, and degraded paths stay bounded under pressure. The system sheds load deliberately instead of hiding overload behind a passing contract.
Review packets Job ids, versions, proofs, and gates line up with the change. The packet reduces reviewer imagination instead of adding ceremony.

Simulation catches pressure, not just shape

Contracts prove that two sides still agree on the shape of a message. They do not prove the system behaves when every valid message arrives too fast, the provider slows down, the queue fills, or the caller retries at the worst possible moment. A perfectly valid request can still create a production-only failure if the harness never applies pressure.

Backpressure simulation belongs in the loop because it turns "what happens when the caller keeps going?" into executable evidence. Inject bursty users, slow providers, saturated queues, retry storms, partial outages, and cancellations. Then check what is rejected, what is deferred, what is retried, what receives a receipt, and what the operator can see while the system is under load.

This layer spots issues contracts are blind to: work accepted after cancellation, orphaned jobs, stale receipts, a retry loop amplifying an outage, one large task starving the rest of the queue, or a UI that says "done" while the backlog is still growing. The goal is not theater-grade load testing. The goal is to make overload behavior reviewable before production traffic becomes the reviewer.

The toaster test

Imagine a toaster factory with no closed loop. The machine pushes bread through heat, the tray fills up, and quality control discovers the burnt pieces afterward. The process can still produce toast, but the control method is late, sensory, and human. Someone has to notice the defect after value has already been damaged.

That is how a lot of AI and agent review works. The system emits a patch, note, answer, or workflow result. Then a person tastes the toast. They ask whether the ids line up, whether the version is stale, whether the job reference exists, whether the tool call had an owner, whether complexity crept up, and whether the evidence supports the conclusion. The questions are valid. The timing is the problem.

Six Sigma's DMAIC loop is useful here because it refuses to stop at taste. Define the defect, measure the current process, analyze the cause, improve the mechanism, and control the future process so the same defect does not keep returning. In software, the control phase becomes the harness: contract checks, property generators, complexity gates, and review packets that keep the process from drifting back into human inspection theater.

Contract tests ask: do we still agree?

Contract testing is useful because it narrows the question. Instead of deploying the whole world to see whether two applications still communicate, a contract test checks whether the messages exchanged by a consumer and provider still match a shared understanding. Pact describes this as testing each side of an integration point in isolation against a contract.

The deeper lesson is not limited to HTTP. Every tool call, review packet, background job, local model server, and handoff artifact has a consumer and provider. The agent provides a job receipt. The reviewer consumes it. The deploy script provides a revision. The live verification consumes it. The contract is the smallest shared shape both sides can depend on.

Property tests ask: what must never stop being true?

Property-based testing moves the center of gravity from example to invariant. Instead of checking only one hand-picked case, the test describes a rule and lets the generator search the input space. Hypothesis frames this as writing tests that should pass for all inputs in the described range while the tool chooses cases, including edge cases the author may not have imagined.

That maps cleanly to agent systems. The interesting properties are rarely exotic: every artifact has one owning job, every submitted review packet has a version, no destructive tool call lacks an owner, no approval record points at a deleted object, and no "done" state appears without proof. Those rules are not just tests. They are the product's ethics, recovery model, and audit spine in executable form.

Constraint Executable property Review receipt
Unique id, version, and job link No duplicate active artifact; version moves forward; job id exists. job_id:artifact_id:vNN appears in packet and logs.
Tool-call preflight Object, action, reversibility, proof, and owner are named before execution. Reviewer can diff the planned action against the completed action.
Complexity budget Branch count, handler size, or decision table growth stays under review limit. Packet flags the exact function or route that crossed the budget.
Consumer-provider contract Request, response, event, or artifact shape still satisfies the known consumer. Contract id, commit, and provider verification sit beside the change.

Complexity constraints are also properties

Complexity budgets often get treated as style preferences. They are stronger than that. A complexity cap says the system should not keep hiding more decision paths inside one unit of code. ESLint's complexity rule makes this literal by warning when cyclomatic complexity crosses a configured threshold.

That is a property. It says the human reviewer should not be asked to simulate an unbounded decision tree during review. When a path crosses the budget, the harness can force the right conversation: split the handler, move the policy into a table, add contract coverage, or attach a review packet explaining why the complexity is deliberate.

A review packet is the harness snapshot

Review packets matter because they convert scattered proof into a bounded object. A good packet does not ask the reviewer to become a detective. It tells them which job ran, which artifacts changed, which contracts were touched, which properties passed, which complexity budgets moved, and which risks remain open.

  1. Identifiers: job id, artifact ids, versions, owners, and timestamps.
  2. Contracts: consumer-provider expectations and compatibility checks.
  3. Properties: invariants checked across generated or enumerated cases.
  4. Complexity: branch, file, route, or decision-table budgets that changed.
  5. Evidence: commands, routes, logs, screenshots, source links, and live receipts.
  6. Open-loop risk: anything still depending on human inspection instead of a gate.

The harness should reduce reviewer imagination

A reviewer should still use judgment. They should inspect intent, product fit, and the shape of risk. But they should not have to imagine whether ids are unique, whether the version moved forward, whether the job link exists, whether the contract still matches, or whether the branch count exploded. The harness should answer those questions before review begins.

This is the practical bridge between testing and operations. Contract tests protect agreements. Property tests protect invariants. Complexity constraints protect human comprehension. Review packets protect handoff. Layered together, they turn "please review this" into "here is the bounded system state, here are the constraints, here is the correctness evidence, here is the quality evidence, and here is what still needs human judgment."

The operating rule:

Every repeated review question is a missing property. Name the question, turn it into a constraint, attach it to the job id, and make the next packet prove it. If a layer keeps asking for the same human judgment, promote that judgment into the next objective-function signal.

Related routes and sources