Hive

Run hive-bench on your machine

A benchmarkable task is a completed Hive coding task in 9-done with a merged GitHub pull request. The merged PR is the held-out reference; candidates receive the frozen task inputs but never that patch.

You need Docker, Ruby 3.4, authenticated agent CLIs for the candidates and judges you select, and enough provider quota for complete planning, implementation, review, and judging runs.

Select Hive’s built-in workflow

Install Hive, then initialize a local benchmark project with the named workflow. You do not need to copy a workflow descriptor or clone hive-bench to select it.

Release status: the built-in bench workflow is currently available in Hive PR #734, not in the latest v0.4.1 release. Use a source checkout of that PR until a release containing it is published.

hive init /path/to/benchmark-project --workflow bench
hive new <project-name> "benchmark my task"
hive status

The normal Hive daemon owns stage ordering, locking, retry behavior, and per-project concurrency. Hive stores the versioned benchmark harness in .hive-state/bench-runtime; no separate hive-bench checkout is needed to run the campaign.

Extract your completed task

Create the corpus entry from the completed task and its merged reference PR:

cd /path/to/benchmark-project
ruby .hive-state/bench-runtime/harness/extract.rb \
  --task-dir /absolute/project/.hive-state/stages/9-done/<task-slug> \
  --repo owner/repository \
  --repo-path /absolute/project \
  --out corpus

Keep the generated corpus entry local when the task is private. Candidates do not receive its held-out reference.patch.

Pre-register and run the campaign

Copy the packaged example into the new task folder reported by hive status, then edit it as campaign.yml. Pin the source repository, corpus tasks, candidate ids, judges and reasoning levels, sample count, timeout, budget declaration, and exclusions before any model spends tokens. Commit that file inside .hive-state; start a new campaign instead of amending it after generation begins.

cp /path/to/benchmark-project/.hive-state/bench-runtime/campaign.yml.example \
  /path/to/benchmark-project/.hive-state/stages/1-inbox/<campaign-slug>/campaign.yml

git -C /path/to/benchmark-project/.hive-state add \
  stages/1-inbox/<campaign-slug>/campaign.yml
git -C /path/to/benchmark-project/.hive-state commit \
  -m "Pre-register benchmark campaign"

hive approve <campaign-slug>
hive run <campaign-slug>

You may run stages manually or enroll the project in the Hive daemon. One campaign task processes its matrix in stable order; independent campaign tasks can run concurrently within Hive’s configured limits.

For the campaign contract, candidate profiles, and evidence format, see the benchmark workflow reference.

Submit a task to the public benchmark

Public submission adds a reusable task to the corpus. It does not spend model quota or immediately change the leaderboard.

hive bench submit <task-slug>

# Disambiguate the project when the slug exists more than once:
hive bench submit <task-slug> --project <project-name>

The task must be in 9-done, include worktree.yml and pr.md, belong to a registered Hive project with a GitHub origin, and point to a merged reference PR. Review the task for confidential material first: its specification, plan, provenance, and merged reference diff become public.

Prepare the submission checkout

Running a local campaign uses Hive’s packaged .hive-state/bench-runtime and does not require a hive-bench clone. Public submission does: use a separate, clean checkout whose origin is writable—normally your fork—and point HIVE_BENCH_PATH at its root.

git clone git@github.com:<your-user>/hive-bench.git /path/to/hive-bench
export HIVE_BENCH_PATH=/path/to/hive-bench

The command creates its submission branch from the checkout’s current HEAD, so start from an up-to-date default branch and keep the checkout free of unrelated changes.

Submit step by step

When you run hive bench submit <task-slug>, Hive:

  1. Runs hive-bench’s canonical extractor and secret scanner.
  2. Writes the generated entry to corpus/<task-slug>.
  3. Creates a submit-<task-slug> branch in the checkout.
  4. Commits the corpus entry and pushes the branch to the checkout’s origin (normally your fork).
  5. Opens a pull request against ivankuznetsov/hive-bench.

Maintainer-gated CI then validates structure, provenance, candidate-visible answer leakage, secrets, and reference reproducibility before the task can join a public campaign.