Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code

This repository implements the Rust generative compiler, the experimental evaluation, and the mechanization of proofs described in the paper Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code. Its Rust sealor closes partial programs into complete programs, then leverages rustc to obtain compiler feedback and returns diagnostics mapped back to the original input. The LLM layer uses those diagnostics to guide generation and roll back rejected prefixes.

Overview

The repository is split by component:

• crates/generative-compiler: Rust sealor and generative compiler, with their tests.
• crates/generative-compiler-rustc: Wrapper around rustc used by the generative compiler.
• crates/rust-analyzer: Patched rust-analyzer source.
• crates/rust: Patched rustc checkout used to build the compiler wrapper.
• bindings: Python bindings for the sealor and generative compiler.
• llm: LLM inference and evaluation code.
• mechanization: Lean mechanization of Featherweight Rust and the sealor.
• patches: Patches that expose the rustc functions used by the wrapper.

Setup

We have provided a Dockerfile for building a container with all prerequisites. Alternatively, they can be installed manually:

• git
• uv
• Rust 1.95.0
• LLVM 21.x

Install uv with the official standalone installer:

curl -LsSf https://astral.sh/uv/install.sh | sh

Rust can be installed using rustup:

curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile=minimal --default-toolchain=1.95.0

LLVM 21 is packaged for most major operating systems:

apt-get install llvm-21-dev    # Debian 13+, Ubuntu 26.04+
brew install llvm@21           # macOS
dnf install llvm21-devel       # Fedora 44+
dnf install 'llvm-devel-21.*'  # Fedora 43
pacman -S llvm21-libs          # Arch Linux

We also provide a script to build it from source:

git clone https://github.com/llvm/llvm-project.git --depth=1 --branch=llvmorg-21.1.8
./build-llvm.sh

Install

Run the following commands from the repository root.

Build our rustc wrapper:

git clone https://github.com/rust-lang/rust.git --depth=1 --branch=1.95.0 crates/rust
git -C crates/rust apply ../../patches/rust-1.95.0.patch
./build-generative-compiler-rustc.sh

Build the generative compiler, sync the Python environment, and link both compiler executables into that environment:

./install.sh

The script uses uv sync to create llm/.venv; the environment does not need to be activated. It also verifies that both compiler executables are available. Rerun ./install.sh after changing the Rust Python bindings or recreating the environment.

Use

The main entrypoint is the eval.generic_inference module.

uv run --project llm python -m eval.generic_inference --help

You can run inference of the models on datasets with:

export OPENAI_BASE_URL="https://openrouter.ai/api/v1"
export OPENAI_API_KEY="sk-or-..."

uv run --project llm python -m eval.generic_inference \
  --model-name openai:openrouter/qwen/qwen3.5-9b \
  --dataset-name crust \
  --task-id bhshell/src/main.rs \
  --max-tokens 1024 \
  --limit 1 \
  --stream \
  --output-file result.json \
  --rollback-strategy restart_with_error --rollback-top-k 1

Datasets

The current checkout registers the following datasets:

crust

The translation benchmark used in the paper, CRUST-bench, restricted to a subset of difficult problems. Task IDs are paths relative to llm/datasets/crust-dataset/RBench.

uv run --project llm python -m eval.generic_inference \
  --dataset-name crust

rust-new-api

The synthesis benchmark used in the paper, API-Upgrade. Task IDs are paths relative to llm/datasets/rust-new-api-bench.

uv run --project llm python -m eval.generic_inference --dataset-name rust-new-api:test

Models

Any HuggingFace transformers model name can be specified.

For OpenAI-compatible streaming endpoints, use an openai: model prefix. The backend is black-box: it can run with constraining enabled, but rejected tokens are handled only via rollback strategy (no single-token top-k resampling).

export OPENAI_BASE_URL="https://your-endpoint.example/v1"
export OPENAI_API_KEY="..."
uv run --project llm python -m eval.generic_inference \
  --model-name openai:gpt-4o-mini \
  --dataset-name crust

If a provider rejects the temperature field entirely, run inference with --temperature omit.

For OpenRouter-specific routing controls, pass request-body JSON through --extra-body-json. Example: prefer a specific provider ordering.

export OPENAI_BASE_URL="https://openrouter.ai/api/v1"
export OPENAI_API_KEY="..."
uv run --project llm python -m eval.generic_inference \
  --model-name openai:moonshotai/kimi-k2 \
  --dataset-name crust \
  --temperature omit \
  --extra-body-json '{"provider":{"order":["Together","Fireworks"],"allow_fallbacks":false}}'

For providers exposing the OpenAI Responses API, use openai-responses: instead:

export OPENAI_BASE_URL="https://your-endpoint.example/v1"
export OPENAI_API_KEY="..."
uv run --project llm python -m eval.generic_inference \
  --model-name openai-responses:gpt-5.3-codex \
  --dataset-name crust

Evaluation is done via

uv run --project llm python -m eval.generic_eval <input> <output>

The script automatically determines the evaluation function based on the metadata stored inside the file. Note that this evaluation step is on purpose decoupled from inference, such that inference (which requires GPUs) can be run separately from evaluation (which usually only requires CPU).

Paper results

Generate the paper results and figures in two stages. From the repository root, first load secret.sh, which must export the API credentials needed by the configured model providers, and run the inference sweep:

source secret.sh
cd llm
uv run python -m eval.run_scripts.run_inference crust-project rust-new-api

The inference runner resumes existing result files, so it can be invoked again after an interrupted run. It stores all generation outputs in llm/results/. You can download our experimental results directly and place the in llm/results/ from the GitHub Release

Once the required results have been generated and evaluated, build the tables and figures:

bash eval/figures/generate_paper_figures.sh

Generated artifacts are written to llm/figures.

Acknowledgement

This material is partially based upon work supported by the Defense Advanced Research Projects Agency (DARPA) Translating All C To Rust (TRACTOR) program under Agreement No. HR00112590134.