How Harness² Works: A Practical Tutorial

Harness² is easier to understand as two loops around a frozen model. The inner loop improves the harness for the task in front of you. The outer loop remembers what worked so the next task starts with a better improver.

Task-level recursion: Propose–Probe–Compose

The README describes the task-level step as a contrastive Propose–Probe–Compose step. The three verbs map to three concrete stages:

1. Propose. A harness editor proposes edits to the current execution harness.
2. Probe. Candidate harnesses are executed on the same task, so their effects are observed rather than assumed.
3. Compose. A composer combines the candidates into a refined harness using behavioral contrasts between executions.

The important constraint is in the last clause: this step does not depend on ground-truth feedback. It learns from differences in behavior between executions, not from a hidden answer key. That is what makes the method usable in an open-world setting where no verifier is available at test time.

The pipeline figure in the official repository shows the same flow with a reference execution, candidate probes and a final composed harness. The repository's caption describes invariant and domain edits; the README body itself keeps the description at the level of "propose edits" and "behavioral contrasts".

Harnessing-level recursion: the improvement harness

Task-level recursion produces a harness for one task. Harnessing-level recursion is what makes the method recursive across tasks.

After a task, the adopted harness and evidence-supported lessons are written into a persistent improvement harness. That improvement harness guides future refinement. The README describes it as updating a continual improvement harness with adopted harnesses and evidence-supported lessons from each task.

In practice, this means there are two different artifacts to watch:

• the execution harness refined for the current task;
• the improvement harness that accumulates reusable lessons.

If you only inspect the current task's output, you are missing the part of the system that is supposed to compound.

Parallel and sequential recursion

Both modes start with k reference executions of the same base harness. They differ in how they spend the budget.

The README's own summary is the cleanest: "Parallel recursion explores wider; sequential recursion refines deeper."

Parallel is not simply "more"; it changes what the composer can compare. Sequential is not simply "slower"; it changes what the next step starts from. If you are comparing the two, keep the task selection, executor, improver, judge and web access fixed.

Parameters

Two nearby flags are worth knowing because they are easy to confuse with --k:

• --workers N runs N domains concurrently; tasks within each domain stay ordered.
• --seed N shuffles the task stream reproducibly. It does not control model sampling.

Your first run

The smallest useful run is not the most impressive one. Start with one step and a tiny task selection:

harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1

Before spending model calls, add --dry-run to preview task selection:

harness2 --bench lab --k 1 --mode parallel --dry-run

Once that completes, judge and report with the same parameters:

harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

Then move to a breadth run:

harness2 --bench lab --k 3 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1

A reasonable progression is --k 1 to prove the pipeline, then --k 3 --mode parallel to see whether candidate diversity helps. Try sequential when you care about carrying evidence forward rather than sampling more candidates. Change one dimension at a time, and give each experiment its own HARNESS2_RUNS_DIR so results do not collide.

Model-backed runs incur API costs for executor rollouts and improvement stages. The full LAB task stream is large; start with a narrow domain and a small task cap, then scale once the setup and judging path is proven.

Reading a run

Do not compare a k = 1 refinement against a k = 3 judge, or a parallel run against a sequential report. The README requires the same benchmark, executor, subset, --k, --mode and seed across refinement, judging and reporting. A mismatch changes the measurement, even if the command succeeds.

Next steps

• Benchmarks — the exact LAB reproduction sequence and the security setup.
• Components — what the editor can actually change.
• Experiment guide — task partitions, concurrency and reproducibility.