Throwing Away What You Built
I spent much of yesterday pouring my mental energy into designing a better AI-assisted code review process. Then I threw it away.
What I learned is that you shouldn’t build every idea you have. I thought I had found a great way to improve the code review process. When I tried to say exactly what the system would improve, the answer was: not much. It was too complicated for the improvement it offered over an agent skill. I didn’t notice this when I was designing it. I had to step away and stop thinking about it before I could see the full cost of building it.
I did get something valuable out of it – a vocabulary. I’ve reproduced it in full.
Language
The review
Review: One stored, guided examination of one Target. Everything else belongs to a Review. Its
state is active or completed.
Target: A text description of what was reviewed and where to find the code, written by a Coding Agent for another Coding Agent to read. Stored as one string, without named parts.
What is being judged
Finding: The lasting identity of one claimed problem. Its Claim defines it. It collects Comments and carries the human’s Adjudication. Its Claim, Location, and Comments cannot change; its Adjudication may change while its Review is active.
Claim: The short statement a Finding makes, such as “the retry loop can continue indefinitely when the API keeps returning 429 responses.” Required and cannot change. Without a Claim there is no Finding.
Location: A readable string pointing to a Finding or a fact cited in Rationale, such as
client/retry.py:84-103.
Comments and Adjudication
Comment: One structured machine contribution about a Finding. Comments can only be appended, never edited or deleted on their own; a machine that changes its mind appends another.
Rationale: Prose that explains a Comment’s conclusion, including any concrete facts and Locations it cites. Stored in one optional text field on Comment.
Adjudication: The human reviewer’s judgment of a Finding, recorded as is_valid plus, when the
Finding is judged valid, an Attention Level. Only the latest Adjudication is kept. It needs no
rationale and overrides every Comment.
Valid: A Finding is valid when the human or AI judging it believes its Claim holds in the real context of the code. This is ultimately a judgment call. A Finding still awaiting human review is Pending, not invalid.
Pending: A Finding with no current Adjudication.
Attention Level: How much attention the reviewer recommends the author of the code give a valid
Finding: high, medium, low, or dismissed. On a Comment it is advice; on an Adjudication it
has the final say.
Remediation Risk: The judgment of the chance that resolving a valid Finding will cause an
unwanted outcome, combined with how harmful that outcome would be. It considers the remediation’s
blast radius, complexity, and uncertainty about the code and its expected behavior. It can be
high, medium, or low. A low-risk remediation usually has a clear, contained change; a
high-risk remediation may affect behavior, callers, data, interfaces, or shared assumptions.
Dismissed: An Attention Level recommending no action for a valid Finding. On a Comment it is machine advice and does not imply human review; the Finding remains Pending until a human adjudicates it. On an Adjudication it records the human’s recommendation.