Harbor

The harbor benchmark runs local dataset in Harbor task format. Point it at a dataset directory, or pass a Harbor job config, and AgentCompass loads every task, starts its image, runs the agent through the selected harness, and scores the result with the task’s Harbor verifier. No per-dataset adapter is needed.

Verification follows Harbor’s own Verifier, so a task scores the same here as under harbor run. The one difference is how failed tasks are averaged; see Scores.

Dataset layout

Tasks come from dataset_path, which works like harbor run -p, and from the datasets and tasks entries of a job config. A dataset directory contributes every subdirectory that Harbor accepts as a task; other subdirectories are skipped.

Each task directory needs task.toml, instruction.md and an environment/ directory. Unless verification is disabled, it also needs tests/test.sh, even when a verifier class replaces it, as in Harbor. Harbor’s environment/, solution/ and other files can stay. AgentCompass does not build environment/Dockerfile, and it does not run any other file under environment/.

How it works

1. Image and working directory. Each task runs its own environment.docker_image. A setup.image in --env-params replaces it for every task in the run. A task without docker_image fails before anything starts, unless setup.image is set. The working directory comes from environment.workdir, or from the literal last WORKDIR in environment/Dockerfile unless setup.image replaced the image.
2. Readiness. If the task declares [environment.healthcheck], the check runs until it passes, using Harbor’s Docker-style retry rules. It runs before the agent starts.
3. Agent. The agent receives instruction.md. Its timeout is the task’s agent.timeout_sec, adjusted by the job config as in Harbor. Execution parameters such as run_timeout_multiplier still apply on top.
4. Verification. The benchmark clears /tests and /logs/verifier and uploads the task’s tests/ to /tests. It then makes /tests/test.sh executable and runs it directly, so its shebang selects the interpreter. Stdout and stderr are merged into /logs/verifier/test-stdout.txt.
5. Rewards. The exit status of test.sh is ignored. /logs/verifier/reward.json is read first, then reward.txt. Every value in reward.json must be a finite number.

A task with [verifier] environment_mode = "separate" or a [verifier.environment] table is verified in a fresh environment. The agent’s /logs/artifacts and any declared [[artifacts]] are restored into that environment first.

Scores

The primary metric score uses Harbor’s primary-reward rule:

Rewards	score
Contains reward	that value
Exactly one key, for example {"accuracy": 0.8}	that value (0.8)
Several keys, none named reward	none: the task completes without a score

All reward keys are kept in the task result under extra.eval_raw_data.rewards. A missing, empty or non-numeric reward file is an evaluation error.

Harbor counts a failed trial as 0 in its mean, but AgentCompass averages only tasks that were evaluated. Compare with Harbor by also checking the error count.

Job config

job_config takes a Harbor job config, the file you pass to harbor run -c. Give a YAML or JSON path, or the same content inline. The installed Harbor package validates it with its own JobConfig model, so the accepted schema always matches your Harbor version.

datasets:
  - path: /path/to/harbor-dataset
    task_names: ["web-*"]
    exclude_task_names: ["web-flaky-*"]
timeout_multiplier: 1.5
verifier:
  import_path: my_verifier:MyVerifier
  env:
    JUDGE_MODEL: my-judge-model
environment:
  override_cpus: 4

Settings are layered as in Harbor, from lowest to highest priority:

1. The task’s task.toml.
2. The job config, which plays the role of Harbor’s job file.
3. The run’s own --env-params and --execution-params, which play the role of Harbor’s command-line flags.

AgentCompass applies these job config fields:

Field	Effect
datasets[].path, task_names, exclude_task_names, n_tasks	Select tasks from a local dataset. task_names that match nothing fail, as in Harbor.
tasks[].path	Add one local task directory.
verifier	Configure verification. See Verifier.
agents[0].override_timeout_sec, max_timeout_sec	Replace or cap the task’s agent.timeout_sec.
agents[0].env	Environment variables for the agent phase.
environment.override_cpus, override_memory_mb, override_storage_mb, override_gpus	Replace the task’s resources.
environment.env	Environment variables for the whole task. They override the task’s [environment.env].
timeout_multiplier, agent_timeout_multiplier, verifier_timeout_multiplier, environment_build_timeout_multiplier	Scale the corresponding timeouts, as in Harbor.
artifacts	Files collected after the agent finishes, in addition to the task’s [[artifacts]].
extra_instruction_paths	Files whose text is appended to every instruction.

Some fields would change what a result means, so AgentCompass rejects them before any environment starts:

• n_attempts other than 1. Use the execution setting k instead.
• More than one entry in agents. A run uses one harness.
• datasets or tasks that name a registry, package or Git source instead of a local path.
• install_only: true.

AgentCompass reports every other declared field as not applied, and the run continues. These include agents[].name and model_name, environment.type, n_concurrent_trials and retry. The run’s own harness, model, --env provider and execution settings choose them instead.

Verifier

The verifier section configures verification for every task in the run:

Field	Effect
import_path	module:Class of a Harbor BaseVerifier to use instead of tests/test.sh. The class must be importable in the AgentCompass process; see Installing a verifier plugin.
kwargs	Keyword arguments for that class.
env	Verifier environment variables. They override the task’s [verifier.env].
override_timeout_sec / max_timeout_sec	Replace or cap the task’s verifier.timeout_sec. Harbor’s default is 600 seconds.
include_logs / exclude_logs	Passed to the verifier, as in Harbor.
disable	Skip verification. Tasks complete without a score.

An imported verifier receives a Harbor Task for the task directory and a TrialPaths directory under the run output, harbor-trials/<task_id>/<uuid>/. It also receives an environment adapter that runs its commands in the AgentCompass sandbox.

The verifier class itself runs on the host, as in Harbor. Its commands, such as tests/test.sh, run in the sandbox through the adapter, and it downloads /logs/verifier into its TrialPaths before reading rewards. Any host-side step, such as calling an LLM judge, needs network access from the host rather than from the sandbox.

Installing a verifier plugin

As in Harbor, install the package that provides import_path yourself, into the Python environment that runs AgentCompass rather than into the task image:

uv pip install --python "$(command -v python3)" /path/to/verifier-project

The class is imported while tasks load. A missing package, or a class that does not subclass Harbor BaseVerifier, fails the run before any environment starts. The default tests/test.sh verifier needs no host package.

Harbor has no per-task import_path. A dataset that needs different scoring per task should do so in each task’s tests/test.sh, or use one verifier class that reads each task’s own files to decide how to score it. Datasets that cannot do either should be split into separate runs.

Environment variable templates ${NAME} and ${NAME:-default} in the task’s [verifier.env] and [environment.env], and in the job config’s env maps, resolve from the host. A missing variable without a default fails the task, as in Harbor.

Using mirrored images

environment.docker_image in task.toml is the only per-task image source. If your environment provider cannot reach the dataset’s registry, copy the images to a registry it can reach and change docker_image in each task to the mirrored name. Keep the digest so the image stays pinned:

[environment]
docker_image = "registry.example.com/mirror/my-dataset:my-task@sha256:<digest>"

Edit a copy of the dataset, not the upstream checkout, so the original names remain available.

Parameters

Pass a JSON object via --benchmark-params '{...}', or put it in the benchmark.params block of a --config YAML. See Benchmark overview for merge precedence.

Parameter	Type	Default	Description
dataset_path	string	none	Local Harbor dataset directory, like harbor run -p. Required unless job_config lists datasets or tasks.
job_config	string or object	none	Harbor job config, as a YAML/JSON path or inline. See Job config.
dataset_name	string	none	Name of the dataset. It replaces the Benchmark id in the result directory name, so runs of different datasets are stored apart, and it is recorded with each task.
judge_model	object	none	Judge model spec , as for other judged Benchmarks. Only id is required; an unset base_url, api_key or api_protocol reuses the tested model’s. The resolved spec reaches the verifier as the JSON variable AGENTCOMPASS_JUDGE_MODEL. It overrides the task’s [verifier.env]; job-level verifier.env overrides it.

sample_ids selects exact task directory names from the resolved tasks and fails if one is missing. Task directory names must be unique across all datasets and tasks. The metric contract declares scalar score as the primary observation, so use avg for k>1. See Metrics and Aggregation.

A standalone run stores results under a directory named after the model, the Benchmark and the harness. Every Harbor dataset runs under the same harbor Benchmark id, so set dataset_name to keep each dataset’s results apart. For example, with "dataset_name": "my-dataset", results go to:

<results_dir>/<model>_my-dataset_<harness>/<run_id>/

Reuse looks for earlier runs in the same directory, so a run reuses only results of the same dataset_name.

Not supported

• Multi-step tasks ([[steps]]) are rejected when loading.
• Building environment/Dockerfile or environment/docker-compose.yaml. Every task needs a prebuilt image in its environment.docker_image. For a dataset that ships only Dockerfiles, build and push the images first, then add docker_image to each task.
• Running the agent or verifier as agent.user / verifier.user. Commands run as the image’s default user. The loader reports these fields as not mapped.
• Running solution/solve.sh through an oracle harness.

Run example

agentcompass run \
  harbor \
  openhands \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "dataset_path": "/path/to/harbor-dataset",
    "sample_ids": ["hello-world"]
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

A run driven by a Harbor job config file, scored by an installed verifier class and an LLM judge. Each task names its prebuilt image in docker_image:

agentcompass run \
  harbor \
  openclaw \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "job_config": "/path/to/job.yaml",
    "judge_model": {
      "id": "your-judge-model",
      "base_url": "https://your-judge-endpoint/v1",
      "api_key": "${JUDGE_API_KEY}",
      "api_protocol": "openai-chat"
    }
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"