How noisy is it?
There is no number on this page yet, and that is the honest state.
The harness, the corpus format, and the build gate ship in 1.2 and are tested.
No repository has been labelled. owlwarden bench
exits 2 saying so rather than printing a figure.
Why not score against our own fixtures
The should-not-fire corpus is a pass/fail gate: it is silent, or the build is red. It does not produce a number, it was written by the same people who wrote the rules, and it therefore measures internal consistency rather than real-world behaviour.
Publishing a precision figure derived from it would be measuring the rule authors' model of the world against itself and calling the result evidence. A dishonest benchmark on a project whose entire positioning is honesty is not a marketing risk - it is a category error.
What the corpus will have to satisfy
A corpus is only worth what its discipline is worth, so the rules are enforced when the file loads rather than trusted:
- Two named reviewers per repository. A single-reviewer label is not a label.
- Disagreements are recorded, not resolved. They are excluded from both the numerator and the denominator. A corpus that quietly resolves its hard cases has optimised away exactly the cases that matter.
- Pinned by commit SHA, with the licence recorded, so a result is reproducible in five years and offline.
- Every label carries a note. A verdict nobody explained is a verdict nobody can check.
- Repositories are added in their own pull request, never alongside a rule change. That is how a benchmark becomes a rubber stamp.
What it will publish
Precision and recall per rule and overall; the false positives listed with path
and line, because a number nobody can inspect is a number nobody trusts;
authenticated precision as its own figure, because it is the
metric most likely to be embarrassing; and the corpus size, repository count,
label count, and disagreement count next to the numbers - a precision figure
without a denominator is marketing.
Precision is a build invariant: a change that drops it below the threshold in
bench/thresholds.toml fails. Raising a threshold is a normal
change; lowering one without a note explaining what was traded fails
the build.
npx owlwarden bench # the full corpus
npx owlwarden bench --rule sql-injection
npx owlwarden bench --format json --out bench/latest.jsonKeep reading
- owlwarden turnWhich of these did I just do?
- RulesTrigger, confidence, and framework-specific fix.
- Agent configHooks, MCP servers, instructions, and editor tasks.
- owlwarden vetCheck a repository before you open it.
- owlwarden sealNotice when your agent's execution surface moves.
- ExposureWhich of these findings do I fix first?
- RuntimesDoes this fix run on Bun, Deno, or Workers?
- CoverageMapped rules and categories static analysis cannot cover.