THE PASS VERDICT
A security report landed in my inbox in February of this year. Forty-three pages. Reproducible build hashes. A machine-generated severity matrix with color-coded rows. At the top of the executive summary, in a font the team had clearly paid a designer to pick, a single verdict: PASS. The contract held $140 million in user deposits.
Three weeks later, working through the callback graph by hand, I found a reentrancy vector in a path the report had classified as informational. Not critical. Not high. Informational. The tool that cleared the contract publishes a confidence score. It does not publish a false-negative rate. Those are not the same number, and an entire bull market is currently treating one as a proxy for the other.
I have been auditing contracts for twelve years. I have watched this industry substitute process for understanding in four distinct cycles: the whitepaper in 2017, the token badge in 2018, the KYC stamp in 2019, and now the machine-generated PASS in 2026. Each substitution lowered the cost of trust and raised the cost of being wrong. This one is structurally different. A human auditor who misses a bug can be deposed, cross-examined, discredited. A model that misses a bug degrades a metric nobody publishes. That asymmetry is not a security improvement. It is a liability transfer with no counterparty on the other side.
Logic does not bleed, but it does break. And right now, at the peak of the loudest bull market since 2021, the industry has decided that a probability is the same thing as a proof.
CONTEXT: HOW THE AUDIT MARKET GOT HERE
The audit market in 2026 is not what it was when I entered it. In 2017, at 31, I joined a boutique firm during the ICO peak and spent three weeks dissecting the ERC-20 implementation of the Zeek Token sale contract. There were maybe forty credible auditors globally. We competed on findings, not on throughput. When I found a critical integer overflow in the claimRewards function that fifteen more senior developers had missed — a groupthink artifact, not a skill artifact — I published an emotionless report on GitHub and refused to soften it. That report prevented an estimated $5 million loss. It also taught me something that has aged extremely well: audit quality is a function of adversarial attention, not of headcount or tooling.
The market has since inverted that premise. Capital allocators want speed. Launch teams want a GitHub badge. Exchanges want a document to point at when they list. The deliverable that satisfies all three is not a deep audit. It is a fast, legible, defensible-looking artifact. AI auditing tools were not the cause of that demand. They were the fulfillment of a demand that already existed.
The plumbing underneath this is not complicated. Between 2023 and 2025, three forces converged. First, LLM code-completion and reasoning models became genuinely useful at reading Solidity, which created a false equivalence between "can read code" and "can reason about adversarial code." Second, the cost of a manual audit rose as senior auditors moved in-house or into protocol work, thinning the independent supply. Third, regulation-by-enforcement — the SEC's preferred modality — made a paper trail more legally valuable than a correct one. A documented audit, in 2026, is frequently a compliance artifact rather than a security artifact. Nobody says this at conferences. Everyone behaves as if it is true.
The convergence produced a product category: the automated audit platform. There are roughly a dozen with institutional traction. They raise at nine figures. They publish marketing that quotes "98% recall on historical vulnerabilities." They integrate with CI pipelines. They issue PASS or FAIL. And in a bull market, when capital is impatient and every week of delay costs a token launch its allocation window, they are being used at scale on contracts that hold real money.
I am not against automation. I use static analyzers. I use symbolic execution. I use fuzzers that run for weeks. The distinction I am making is narrow and, I think, unanswerable: automation that outputs an opinion is not the same as automation that outputs a proof, and the industry is deploying the former while advertising the latter.
CORE: A SYSTEMATIC TEARDOWN
WHAT THESE TOOLS ACTUALLY DO
Strip the branding and you find four distinct components, frequently bundled and rarely disclosed separately.
Static analysis is the oldest layer: pattern matching over the AST and control-flow graph. Slither-class tooling. It is deterministic, fast, and blind to anything that requires understanding intent. It will catch a tx.origin authentication flaw. It will not catch a logic error that is syntactically clean.
Symbolic execution and fuzzing form the second layer: property-based exploration of reachable states. This is where real value lives, and it is also where the tooling exposes its own assumptions — you must supply the properties, and if you supply the wrong properties, the tool proves a theorem about the wrong system.
The third layer is the LLM triage and summarization stack: a model reads the findings from layers one and two, deduplicates them, scores severity, and writes the prose. This layer determines what a human sees. It is also the layer with no formal guarantees whatsoever.
The fourth layer is the narrative layer: the PASS/FAIL verdict, the confidence score, the executive summary. This is a product decision, not a technical one. Nothing in the underlying analysis produces a binary verdict. The verdict is a UX choice, and it is the most consequential line in the document.
When I asked one vendor, directly, what a confidence score of 0.94 meant in their product, the answer — after two escalations — was that it measured the model's self-assessed certainty, not the probability that the contract was secure. Those are unrelated quantities. The first is a language-model output. The second is a fact about the world. Presenting the first in the position where a reader expects the second is not a bug in the report. It is the report.
THE TRAINING DATA PROBLEM
Here is where my 2025 white paper on automated auditing intersects with reality. The models are trained largely on public audit reports, public postmortems, and public exploit repositories. That corpus has three defects that transfer directly into the model.
Survivorship bias. Public reports are the ones clients agreed to publish. Clients decline to publish reports containing findings they consider embarrassing. The corpus is therefore enriched in well-behaved audits and starved of the ugly ones. The model learns what a competent audit looks like, not what a dangerous contract looks like.
Label bias. A finding is labeled "critical" because a human auditor, in 2021, under commercial pressure, called it critical. Severity in this industry is negotiated. The model inherits the negotiation without the context. It will confidently mark a non-issue as high and an existential flaw as informational, because that is what the training labels did.
Temporal bias, and this is the one that kills. The corpus is historical. Exploit techniques are not. In 2017 the dominant bug class was integer overflow, which Solidity 0.8.x made largely obsolete by reverting on overflow. In 2020 it was oracle manipulation. In 2022 it was economic-design failure. In 2024 and 2025 it was compiler-version-specific behavior — optimizer bugs, transient storage semantics, memory-safety assumptions in inline assembly. A model trained on 2018–2023 data is structurally incapable of recognizing a 2026 vulnerability, because the 2026 vulnerability did not exist in its training distribution. It will not flag it. It will not express uncertainty about it. It will render a confidence score for a question it does not understand.
This was precisely the flaw I identified in the AI-driven audit tool used by a major firm in 2025. The tool had been trained on historical compiler vulnerabilities and then deployed against contracts compiled with a toolchain released after its training cutoff. The industry dismissed the concern as Luddite fear. Subsequent breaches, in which the tool cleared contracts containing exactly the class of bug it had never seen, were reported as isolated incidents rather than as the predictable output of a system trained on the past to certify the future. Bias hides in the assumptions, not the syntax. The syntax the model saw looked fine. The assumption — that the vulnerability set is stationary — was wrong, and it was wrong by construction.
THE CONFIDENCE-INTERVAL DEFECT
This is the deepest problem and the one least discussed, because it is a problem in mathematics rather than in engineering, and engineers are uncomfortable disliking mathematics.
A security audit is asking a question about a set of reachable states. "Is there any sequence of transactions, under any adversarial strategy, that violates an invariant?" The correct answer to that question is a proof or a counterexample. It is not a probability, because the question is not random. The adversary is not sampling from a distribution. The adversary is searching, adversarially, for the worst path. A probability estimate over an adversarial search is meaningless in the same way that a weather forecast is meaningless to a sniper who can choose the day.
The tooling confuses this in a specific way. Models produce calibrated token probabilities. Those probabilities describe the model's uncertainty about its own next token. They do not describe the probability of an exploit existing. Mapping one onto the other is a category error with a friendly user interface. Yudkowsky-adjacent people have written about this for years in the general case. In crypto it has a concrete consequence: a PASS verdict at 0.97 confidence and a PASS verdict at 0.60 confidence both read as PASS, and the difference is invisible to the person writing the check.
Volatility is just unaccounted-for variables. A confidence score is an accounting device that records which variables the model accounted for. It says nothing about the variables it did not.
COMPILER-VERSION BLIND SPOTS, IN DETAIL
Let me get concrete, because abstraction is where accountability hides.
Between 2022 and 2025, the Solidity compiler introduced — and in several cases later patched — behavior that changed the security surface of contracts without changing a single line of source. Optimizer bugs that produced incorrect code from correct source. Changes to how memory-safe assembly is inferred, which changed which functions got optimized differently, which changed gas edge cases, which changed reentrancy exposure at the boundary. Transient storage, introduced via EIP-1153, which added a new state lifetime that most audit checklists had no category for. The cancellation of a specific optimizer sequence after a bug report affecting a narrow version range.
Each of these is a class of vulnerability that does not exist in the source code and therefore cannot be found by reading the source code. It exists in the interaction between source, compiler version, optimizer settings, and bytecode. A model trained to read source — which is nearly all of them — is reading the wrong artifact. The code speaks louder than the whitepaper, but the bytecode speaks louder than the code.
I have watched a tool certify a contract as clean, and then watched a second tool, given the same contract with a different compiler setting, flag it as containing a known optimizer-adjacent miscompilation. Same source. Same model. Different pragma. The PASS was a statement about a configuration that had not been the one deployed.
The practical fix is unglamorous: the tool must be bound to the exact compiler version, the exact optimizer runs, the exact bytecode hash, and it must refuse to certify anything it has not seen a training example for. No vendor I have reviewed does all four. Some do one.
THE SEVERITY MATRIX AND ITS INCENTIVES
Severity scoring in automated reports is often presented as objective. It is not. It is a compression of a commercial relationship into four colored rows.
A real audit has a triage conversation. The auditor asks the team what the invariant is. The team explains the intended behavior. The auditor tests whether the code enforces the intended behavior. Severity emerges from that dialogue — a finding is critical because it breaks an invariant the team actually cares about. Automated tools do not have that dialogue. They assign severity by pattern, and the pattern was learned from reports written about other systems with other invariants.
The consequence: an AI report will mark a real reentrancy as informational because in its training data, reentrancy in a contract that also had a guard elsewhere was classified as informational — and it cannot see that the guard here is bypassable. It will mark a non-issue as critical because the surrounding tokens match a known exploit template. The report looks rigorous. It is a retrieval system wearing a lab coat. Aesthetics are often exploits in waiting, and a beautifully formatted report is a very effective aesthetic.
INSURANCE THEATER AND LIABILITY TRANSFER
Here is the part that should worry allocators more than users, because allocators are the ones who will hold the bag.
An audit report has become an input to two downstream decisions: whether to list, and whether to insure. Listing desks and insurance underwriters — the actual professional risk-takers in this market — read the PASS verdict, not the appendix. When a tool produces a PASS, the underwriter's model ingests a boolean. The boolean is treated as evidence of absence of vulnerability.
It is evidence of absence of detected vulnerability. The distinction is enormous and it is now priced at zero. Trust is a vulnerability vector, and the most efficient way to attack it is to make the trust signal cheap to produce and expensive to falsify. A system in which the trust signal is generated by a model that degrades invisibly is exactly that.
The insurance wrapper does not reduce system risk. It relocates it from the user to the underwriter, and the underwriter has priced the relocation using a false premise. When the first large loss arrives that was cleared by an AI PASS, the correction will not be gradual. Underwriters will withdraw from the category, listings will tighten, and the same tools will be rebranded as "assistive." I have seen this movie. It played in DeFi Summer and again in Terra.
THE EMPIRICAL RECORD, HONESTLY STATED
In fairness: these tools do catch things. Slither catches real bugs. Fuzzing finds real edge cases. LLM triage meaningfully reduces the time a human spends on noise. I have used all of this and my own throughput went up.
What I have not seen is a published, audited, adversarial evaluation of false negatives. Every vendor publishes recall on a benchmark. Benchmarks are constructed by vendors or by parties with an interest in the tools looking good. Nobody publishes the contracts the tool cleared that were later exploited. That dataset is scattered across postmortems and legal filings and is not aggregated, because aggregating it would be an advertisement for the problem.
In 2021, I audited a generative art project — I will call it CryptoPeas — that raised $2 million. The randomness function used blockhash, which is predictable and exploitable. The team dismissed the finding as "a feature, not a bug." A bot attack drained 40% of the liquidity. The community's response was not to examine the code. It was to defend the art. The lesson I took was not about blockhash. It was that a security finding, no matter how technically clean, loses to a narrative the community has already adopted. An AI PASS is that narrative in a more efficient format.
THE CROSS-CHAIN POSTSCRIPT
While I have the floor, a related observation because it lands in the same accounting: the Dencun upgrade lowered the cost of moving data between rollups, and the industry has treated that as a UX win. It is not. Withdrawing from a centralized exchange remains, in 2026, orders of magnitude simpler than moving assets across two L2s with different finality assumptions, a bridge that may or may not be audited, and a withdrawal delay measured in days. The cost fell. The complexity did not. Complexity is the enemy of security, and a cheap bridge is still a bridge.
This matters because the same psychology that accepts a cheap bridge accepts a cheap PASS. Both substitute a lower cost for a lower risk. Both are purchased by people who have correctly identified the price and incorrectly identified the product.
CONTRARIAN: WHAT THE BULLS GOT RIGHT
I want to be precise about what I am not arguing, because the critique has a blind spot and I should name my own.
The advocates of AI auditing are correct on three points. First, the human auditor baseline is worse than its reputation. The Zeek Token overflow was missed by fifteen senior developers. Human auditors share training data, share tools, share conferences, and therefore share blind spots — groupthink is not a moral failing, it is a network property. A model with a different failure distribution can, in principle, be a useful decorrelation. Second, scale is real. There are more contracts deployed in a week in 2026 than were audited in all of 2017. Manual coverage is arithmetically impossible; something has to do the first pass. Third, the alternative to an AI PASS is frequently no audit at all, and a tool that catches 70% of a bug class is strictly better than a team that audited nothing.
All three are true. None of them justify the PASS verdict.
The correct conclusion from "AI auditing is better than nothing" is not "ship the tool." It is "ship the tool but do not let it issue a binary that downstream systems read as proof." The bulls are right about the tool and wrong about the interface. The false-negative rate, the training cutoff, and the compiler version belong in the executive summary, at the top, where the PASS currently is. Anything less is marketing wearing the clothes of engineering. Every artifact is a trace of failure. A report that hides its own uncertainty is a trace of the failure to disclose.
TAKEAWAY
So here is the forward-looking judgment, and it is a question of accountability rather than of technology. The 2026 bull market will produce at least one loss in the nine figures that was cleared by an AI PASS. When it happens, the postmortem will name the exploit, the contract, and the dollar amount. It will not name the confidence score, because the confidence score was never published. That absence is the accountability gap, and until a regulator or an underwriter requires the score to be disclosed alongside the verdict, the gap will widen as the market grows.
The code speaks louder than the whitepaper. Right now the confidence score is louder than both. That is the variable nobody has accounted for.
