A Harness2 bug report shows an old passing grade attached to a new result
An October 18 issue in Google Research's Harness2 repository reports that a failed evaluation can attach an old score to a replacement result. The reporter reproduced the bug locally. The task runs again and produces new output, but a failed judge call leaves the earlier passing grade in place. Pull request 45 proposes a fix.
Verified 12:20 AM PDT · 2 original sources
The issue and proposed fix remain open. HotTea has not independently reproduced the bug. The report concerns cached evaluation results. It does not show that the bug affected scores published in the research paper.
Watch whether the fix is merged and how it handles older result files. Each score should identify the output that received that grade, especially after a task runs again.
Audit the story
Original sources
Company claims remain company claims. Follow the reporting and judge the evidence directly.
- Harness2 issue reporterOld grade survives a task rerun when the next judge call fails ↗
- Harness2 contributorKeep a cached grade only for the same selection ↗
Continue the edition