EvalBook evidence
How an EvalBook evaluation report becomes evidence on a pull request — fetched by id and re-hashed by Madebook, trusted only as far as EvalBook's own assurance, counted only on the exact commit — how "EvalBook suite N must pass" joins the compliance check, and how EvalBook asks Madebook before it changes a protected suite, scorer, judge, dataset or threshold.
EvalBook measures; Madebook governs. When a workspace is linked to an EvalBook project (see Linked products), two things happen and nothing else: EvalBook's reports become evidence on the workspace's pull requests, and EvalBook asks Madebook before it changes its own protected configuration. An EvalBook pass never approves anything by itself — the compliance check still needs its independent approval, and a governed EvalBook change is still decided by a person's vote or an approved policy.
A report is a lead until Madebook has re-fetched it
EvalBook tells Madebook that an evaluation finished. Madebook never uses the verdict in that message. It asks EvalBook which reports exist for the pull request's head commit, fetches every one of them again by id, and recomputes the report's digest from the bytes it received. A report counts only when all of this holds:
- its content hashes to the digest EvalBook named for it, and to the same digest every time it is fetched (a report never changes after it is issued);
- it is about this organization, this repository and this exact head commit;
- it is in a report schema Madebook reads, inside its validity window, and not superseded;
- a report that says pass does not contradict itself — its required checks passed, its evidence is complete, its coverage lists nothing missing.
Anything else is refused, and the refusal is shown with its reason on the pull request page (a digest that does not match, another commit, an expired window, a self-contradiction). A refused report is never silently missing. EvalBook not answering leaves the commit not established, which fails — never "no report".
Trust comes from EvalBook's assurance, and from nothing else
Every piece of evidence in Madebook has a source: reported, observed or attested (see Evidence and gates). An accepted EvalBook report is recorded as:
| EvalBook's assurance | Source in Madebook | Counts toward "suite N must pass" |
|---|---|---|
platform_verified — EvalBook ran the suite itself |
observed | yes |
| anything else (self-reported, relayed, unspecified) | reported | no — it is shown, labelled as not counting |
No signature promotes a report further, and neither does anything the report says about how it was produced.
"EvalBook suite N must pass"
On the workspace's Compliance → EvalBook tab, anyone who may propose a decision can propose a requirement: EvalBook suite 5 must pass, for every repository in the workspace or one. It is a draft and holds nothing to anything until someone other than its proposer approves it. Approving or retiring one re-evaluates the open pull requests it reaches, as does linking or unlinking the EvalBook project.
Each approved requirement is one item of the compliance check, evalbook:<suite>:
| What there is | Item | The detail says |
|---|---|---|
| the newest current, verified, platform-verified pass on this commit | met | the report, its digest and until when it is valid |
| a verified report that failed, errored or was inconclusive | unmet | which, on which commit |
a pass whose assurance is weaker than platform_verified |
unmet | it stays reported: EvalBook relayed it, did not run it |
| a refused report | unmet | the refusal |
| a superseded or expired report, and nothing newer | unmet | stale or replaced, and by what |
| EvalBook answered with no report for the suite | unmet | no report, and when it was asked |
| no EvalBook link, never asked about this head, or EvalBook did not answer | unknown | which — and unknown fails |
An older pass never outvotes a newer failure on the same commit. A push supersedes EvalBook evidence at once, exactly like CI, and the hourly sweep re-evaluates pull requests whose counted report has expired. Madebook never recomputes a score: it decides whether the evidence EvalBook supplied satisfies the rule for this exact commit.
The pull request page shows the EvalBook panel: the link, the requirements, the last time EvalBook was asked, and every report with its verification. Ask EvalBook now asks again immediately rather than waiting for the next sync.
EvalBook asks before it changes protected configuration
A linked EvalBook asks Madebook before it activates a suite version, changes a pass threshold, activates a scorer version or a judge configuration, edits a protected dataset or a holdout, or stops being governed. Madebook answers allow, deny, require approval or defer — deterministically, from the organization's (and the workspace's) approved evaluation governance policy, never from a model.
The rules that cannot be configured away:
- Nothing defaults to allow. An unknown kind of change, a missing digest or a target that names nothing is denied. With no policy approved, Madebook's default applies: every kind needs one independent approval. Bookbag account or the policy being unreadable is defer — answered, not stored, so EvalBook asks again and does not act meanwhile.
- The authors of a change, its requester and the person EvalBook acted for never approve it, compared by their Bookbag identity. A policy can waive review of somebody else's change, never a person's review of their own.
- An approval is bound to the exact change (its digest). Asking again for the same target with a different digest is a new decision, and the earlier one stops counting.
- An allow is valid for a bounded time; an approval nobody gives lapses to deny; a link that ends denies everything it carried.
Require approval opens an ordinary decision in Questions waiting on you, naming the exact change, its digest, who asked and who wrote it. It is settled by votes from people with the approver role (and in the reviewer team, when the policy names one) — one rejection denies, the policy's quorum allows. It cannot be settled with a single answer like other questions. When it resolves, EvalBook is told and reads the decision back before it activates anything.
The policy itself — one effect per kind of change, the quorum, the reviewer team, the windows — is proposed and approved like any other rule: Organization → Approval rules → Evaluation governance for the organization, and a workspace's Compliance → EvalBook tab for a stricter override. A workspace can only tighten what the organization set.
What has run for real, and what has not
The whole of the above was exercised against an EvalBook stand-in that signs and verifies exactly as EvalBook's contract says, over HTTP, on SQLite and on PostgreSQL: linking, reports fetched by id and re-hashed, a pass counted as observed, a failing and an inconclusive report blocking, a relayed pass staying reported, a tampered report refused, a governed change approved by vote and read back as allow, an unlink denying what it carried. Nothing has crossed the boundary to a real EvalBook yet; the application's docs/PEER-PROTOCOL.md section 7 is the contract both sides build to.