Harness² Benchmark Results

Harness² ships three professional benchmarks. The headline result is deliberately narrow: a single Harness² step improves the model over its base harness, with the model weights fixed. The numbers below are gains, not absolute scores.

The reported result

Read that table with three constraints:

1. The gains are relative to each base harness. They are not absolute benchmark scores and they are not a claim that the model reaches a particular pass rate.
2. They are the reported maximums across three professional benchmarks and four model–harness configurations. "Up to" is not a typical or average result.
3. The models named in the README are GLM-5 and Gemini 3.5 Flash. Do not generalize the figures to other models without measuring them.

The three benchmarks

harness2-setup wb without a subset installs all three subsets in one run. Pass --subset to install only the one you intend to run; every other WorkBuddy command takes one subset at a time.

Reproduce LAB

This is the README's LAB sequence, including the judge and report steps. Run it from the repository root with the Harness² environment active:

harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

The refinement command is deliberately small: one domain, one task. --max-tasks caps tasks per domain, not across the whole run. Add --dry-run to harness2 to preview task selection without model calls.

Judging and reporting must use the same benchmark, executor, subset, --k, --mode and seed as the refinement run. If you change any of them, you are measuring a different run.

harness2-judge --bases scores a run namespace's reference executions and writes base_judged.json beside judged.json for the base-harness comparison. harness2-report covers the refined harness only.

What the parameters mean

The README's own summary is the clearest guide to the tradeoff: parallel recursion explores wider; sequential recursion refines deeper. The tutorial explains both in more detail.

Security: the part most write-ups skip

The release does not restrict network access. Codex runs with its sandbox disabled. All three benchmarks are public, and JobBench publishes each task's RUBRICS.json on Hugging Face. An executor with network access can retrieve grading criteria. The experiment guide reports a validation run of this release in which a Codex rollout on JobBench downloaded its task's rubric without being instructed to.

Before evaluation, block these hosts for the executor:

• huggingface.co, hf.co, and *.hf.co
• github.com and raw.githubusercontent.com

Setup itself needs these hosts to download dependencies and data. Apply the restriction at the network level, such as an egress firewall or DNS policy on the executor host, rather than relying on a tool setting. Report the restrictions you used alongside your results.

If you use Codex, review the web-access section before the run. The bridge and sandbox details are in the setup guide.

Results, resume and comparison

Results are saved under runs/, organized by benchmark and subset where applicable. They include task summaries, deliverables, adopted harnesses, reflection notes and evaluation results. The default roots are runs/lab/, runs/jb/ and runs/wb/<subset>/.

Repeat the same command to resume. Use a new HARNESS2_RUNS_DIR when changing models or task selection; the saved configuration prevents incompatible resumes. For JobBench, setup records the downloaded Hugging Face revision in dependencies/job-bench-eval/dataset/main/.harness2-download.json; report that revision alongside JobBench results.

Next steps

• Tutorial — what --k, --mode and --substrate actually change.
• Install — prerequisites and the dependency traps before a benchmark run.
• Experiment guide — task selection, partitions, baselines and reproducibility.