Install Harness² (harness2)

Harness² is a Python package plus a set of benchmark adapters. The local demo is deliberately boring: it runs both recursion modes with deterministic editors and real Python executions, and it needs no model credentials. Model-backed benchmarks are the part that needs cloud setup.

Prerequisites

The setup guide adds one useful nuance: the Python package also supports Python 3.13, even though the README's quick start uses 3.12. The benchmark host itself is documented as Linux. Windows and macOS are not listed as supported benchmark platforms.

Install from the repository root

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2

The first three lines create an isolated environment and install Harness² with the rendering extra. The fourth line runs the local demo.

What the demo proves

The demo runs both recursion modes with deterministic editors and real Python executions. It requires no model credentials. That makes it the right first check: if the demo fails, the problem is local packaging or Python, not Vertex AI access.

--output runs/demo writes the run artifacts under runs/demo. --k 2 gives the demo a two-step task-level budget so you can see more than a single Propose–Probe–Compose pass.

Configure models for real benchmarks

Copy the example environment file, fill in your Vertex AI project and model IDs, then authenticate:

cp .env.example .env
# Edit .env with your Vertex AI project and model IDs.
gcloud auth application-default login

The setup guide documents these settings:

Shell environment variables take precedence over .env. The provided configuration uses Vertex Gemini executors and judges, and supports either Vertex Claude or Gemini as the improver. LAB also supports the upstream judge's direct providers when their credentials are configured.

If the browser flow is inconvenient, the setup guide lists alternatives: gcloud auth application-default login --no-browser, a service account key through GOOGLE_APPLICATION_CREDENTIALS, or impersonation with --impersonate-service-account. Keep gcloud on PATH in every case, because the installation check runs gcloud auth application-default print-access-token.

Install a benchmark

Benchmarks install separately. Install only the one you intend to run:

harness2-setup lab
harness2-check --bench lab

Use harness2-setup jb for JobBench or harness2-setup wb for WorkBuddy-Bench. Each benchmark environment lives inside its upstream checkout, so do not activate those environments for the main commands; Harness² selects the interpreters automatically.

harness2-check verifies dependencies, credentials and configuration without calling a model. It does not test model availability. Access to an executor, improver or judge model is tested when that model is first called.

Common pitfalls

These are the places where the documented dependencies actually bite. The official docs do not publish these as error messages, so treat this as a map of failure points rather than a transcript.

• No Linux host. The README says Linux. If you are on macOS or Windows, the documented benchmark path does not apply; expect to solve installer and container issues yourself.
• No Python 3.12. uv venv --python 3.12 needs a 3.12 interpreter. uv can download CPython from astral.sh when the host has no system 3.12, so a blocked network can fail here. The package also supports 3.13, but the quick start is written for 3.12.
• Virtual environment not active. After source .venv/bin/activate, harness2-demo and harness2 should resolve. If you skip activation, uv pip install may install into a different environment and the commands will not be on PATH.
• No Node.js 20+. The local demo does not need it. Benchmark setup and the Codex CLI do. Install Node before benchmark work, not before the demo.
• gcloud missing from PATH. The install check calls gcloud auth application-default print-access-token. A working gcloud that is not on PATH still fails the check.
• No Pandoc or Poppler. LAB needs both; on Debian/Ubuntu the setup guide installs pandoc and poppler-utils. Other benchmarks do not.
• Docker not running or not permitted. WorkBuddy-Bench needs Docker Engine and permission to run docker info. A running daemon is not enough if your user cannot talk to it.
• Restricted network during setup. Benchmark setup clones pinned upstream repositories from github.com, fetches packages from registry.npmjs.org, downloads data from huggingface.co and its CDN, pulls a Docker image, and may fetch CPython 3.12 from astral.sh.
• Codex bridge not running. The Codex executor needs the optional bridge dependencies and a running harness2-bridge. For WorkBuddy, start it with harness2-bridge --host 0.0.0.0 so containers can reach it, and pass --substrate cx to setup, checks, refinement, judging and reporting.
• Wheel install expectations. A wheel ships the harness2 package only. The task partitioner, python scripts/split_tasks.py, requires the source tree. For a wheel install, work from a dedicated directory because .env, dependencies/ and runs/ are resolved there.
• If a benchmark fails after the demo succeeds, the cause is almost always a benchmark-only dependency — Node 20+, the gcloud CLI, Docker, or Pandoc and Poppler — not the install itself.

Next steps

• Setup guide — full requirements, credentials and benchmark installation.
• Codex executor — bridge setup and --substrate cx.
• Benchmarks — what to run after installation.
• Tutorial — how --k and the recursion modes change a run.