GenesisBench

GenesisBench evaluates how coding agents can use language intelligence to improve physical intelligence.

A task gives an autonomous coding agent:

• a robotics environment or simulator;
• a fixed starter policy, controller, planner, or training system;
• queryable development feedback;
• a bounded research budget;
• a standardized final-artifact contract.

After the agent exits, GenesisBench independently evaluates its final artifact on a clean, hidden suite and assigns the resulting robotics score to the agent. The workflow is inspired by PostTrainBench, but the optimized artifact controls a physical system rather than being an instruction-tuned language model.

Reference Task: Simulation Heuristics Ant v1

tasks/simulation_heuristics_ant_v1/ is the first executable task and the canonical example for future contributors. An agent receives a weak rhythmic CPG/PD controller for Gymnasium Ant-v5, repeatedly edits and evaluates it, and submits final_policy/policy.py.

The package follows BenchFlow 0.6.5's native task.md format (schema_version: "1.3", document version "0.6").

Final scoring uses full 1,000-step episodes:

score = 0.70 * hidden nominal mean return
      + 0.30 * hidden dynamics-robustness mean return

The checked-in reproducibility suite includes unseen seeds and conservative mass, friction, damping, and actuator perturbations. An official hosted leaderboard can inject a private suite without changing the task contract.

Quick Start

Requirements:

• Python 3.12+
• uv
• Docker for isolated agent experiments

Install and validate:

uv sync --extra dev
uv run python scripts/validate_tasks.py
uv run bench tasks check \
  tasks/simulation_heuristics_ant_v1 \
  --level publication-grade
uv run pytest -q

Evaluate the starter policy:

uv run python tasks/simulation_heuristics_ant_v1/evaluate.py \
  --policy tasks/simulation_heuristics_ant_v1/starter_policy/policy.py

Prepare exactly the public workspace an agent receives:

uv run python scripts/prepare_task.py \
  simulation_heuristics_ant_v1 \
  /tmp/genesisbench-simulation-heuristics-ant-v1 \
  --force

The prepared OpenHands workspace deliberately excludes verifier/, oracle/, and evidence/.

OpenHands Experiment

Build the isolated runner:

sh scripts/build_simulation_heuristics_ant_v1_runner_image.sh

Configure credentials:

cp .env.example .env

Run one agent:

uv run python scripts/run_simulation_heuristics_ant_v1_experiment.py \
  --model gpt-5.6-sol \
  --minutes 30

See experiments/simulation_heuristics_ant_v1/README.md for model routes, fairness controls, artifact layout, and leaderboard regeneration.

Current Leaderboard

The first four-model OpenHands sweep used equal 30-minute budgets and each model's highest supported reasoning setting. Machine-readable results and packaged policies are in leaderboard/.

Rank	Model	Hidden-suite score
1	GPT-5.6 Sol	3417.86
2	GPT-5.5	2382.23
3	GPT-5.4 Mini	2369.61
4	Claude Opus 4.8	2235.71

These are single-run research results, not multi-trial estimates of model quality. See leaderboard/REPORT.md for setup details and limitations.

Contribute a Task

Create a scaffold:

uv run python scripts/create_task.py my_robot_task \
  --title "My Robot Policy Improvement Task"

Then:

1. Read tasks/README.md.
2. Study the complete reference task in tasks/simulation_heuristics_ant_v1/.
3. Implement the starter artifact, public evaluator, and clean final verifier.
4. Run uv run python scripts/validate_tasks.py.
5. Include a real coding-agent canary and reproducible score evidence.

See CONTRIBUTING.md for the full contribution workflow.

Roadmap

• GenesisBench 1.0: language intelligence improves physical intelligence.
• GenesisBench 2.0: world intelligence improves physical intelligence.
• Add manipulation, navigation, whole-body control, data generation, and sim-to-real tasks while preserving task-level resource accounting and clean final evaluation.

Research Background

• Learning Beyond Gradients
• Autoresearch
• Autoresearch Robotics
• Genesis
• MuJoCo
• Isaac Sim
• RoboCasa
• NVIDIA ASPIRE
• NVIDIA ENPIRE

License

GenesisBench is licensed under GPL-3.0. See LICENSE.

Some reference-policy code is derived from Apache-2.0-licensed work. See THIRD_PARTY_NOTICES.md and LICENSES/Apache-2.0.txt.