Evaluation & systems · Local research prototype
AutoSeal
Inspect the evaluator behind the reward.
A local evaluator-integrity prototype that binds candidate execution to retained evidence, compares proxy and independent audit verdicts, and previews repair policies.
The question
A passing process or benchmark verdict can be misleading when candidate-controlled configuration changes which tests execute.
How it works
- Use a typed process runner, content-addressed receipts and separate proxy/audit views.
- Reproduce a disclosed failure on a pinned public case with explicit legitimate, invalid and exploit-bearing controls.
- Require fail-closed review of candidate identity, control roles and preservation checks before considering a repair.
Evidence & results
- On one selected SWE-bench Verified instance, seven native/adapter controls and separate semantic audits are archived.
- The audit accepted two valid fixes and rejected five other controls, including disclosed false-reward variants.
- A separate tiny training diagnostic established a real optimizer update but no difference between paired evaluator arms.
Scope & next steps
- Known disclosed controls, not blind discovery, a benchmark-wide attack rate or general efficacy.
- No repair promotion or downstream model improvement is claimed.
- The local execution setup is not a reviewed hostile-code security boundary; independent personal reconstruction remains open.
Go one level deeper.
Ask about the implementation, tradeoffs, or my contribution.
Explore with the portfolio guide