Evaluation & systems · Local research prototype

AutoSeal

Inspect the evaluator behind the reward.

A local evaluator-integrity prototype that binds candidate execution to retained evidence, compares proxy and independent audit verdicts, and previews repair policies.

The question

A passing process or benchmark verdict can be misleading when candidate-controlled configuration changes which tests execute.

How it works

  • Use a typed process runner, content-addressed receipts and separate proxy/audit views.
  • Reproduce a disclosed failure on a pinned public case with explicit legitimate, invalid and exploit-bearing controls.
  • Require fail-closed review of candidate identity, control roles and preservation checks before considering a repair.

Evidence & results

  • On one selected SWE-bench Verified instance, seven native/adapter controls and separate semantic audits are archived.
  • The audit accepted two valid fixes and rejected five other controls, including disclosed false-reward variants.
  • A separate tiny training diagnostic established a real optimizer update but no difference between paired evaluator arms.

Scope & next steps

  • Known disclosed controls, not blind discovery, a benchmark-wide attack rate or general efficacy.
  • No repair promotion or downstream model improvement is claimed.
  • The local execution setup is not a reviewed hostile-code security boundary; independent personal reconstruction remains open.

Go one level deeper.

Ask about the implementation, tradeoffs, or my contribution.

Explore with the portfolio guide