Introduction
A faster prover is useful only when the benchmark measures proving.
That sounds obvious. In practice, a first run may compile the program, synthesize proving and verifying keys, fetch parameters, populate filesystem caches, initialize Rayon workers, and then generate a proof. Comparing that run against a warm process tells you very little about the proving change itself.
snarkVM PR #3399 is a good reason to build a stricter test rig. One change moves each chunk's seed exponentiation outside the main loop in evaluate_all_lagrange_coefficients. Parallel chunks still need independent seeds because they can execute out of order, but the expensive exponentiation no longer repeats inside the Lagrange evaluation loop.
A microbenchmark can show whether that function became faster. Developers also need an end-to-end number from a real Aleo program, since an isolated win may disappear among circuit synthesis, proof construction, serialization, and verification.
We will measure both. No proof gets counted until it verifies.
What we're building
The project has two layers:
- A fixed Leo 4.4 program containing 64 dependent Poseidon2 hashes.
- A TypeScript runner using Provable SDK 0.11.6 to generate and verify local proofs.
- A shell wrapper that pins the thread count and records operating-system peak RSS.
- A comparison script that rejects invalid runs and fails when warm median time or peak memory crosses a chosen threshold.
The benchmark separates key synthesis from measured proving. Cold means the first proof generated by a new ProgramManager after the keys already exist. Warm means later proofs generated in the same process with the same manager and keys.
Those definitions matter. Deleting proving keys and calling the result a cold proof measures setup plus proving. Starting a new process for every warm sample measures process startup and runtime initialization. Both mistakes can bury a small arithmetic improvement.
The workload also has a state-writing entry point. We do not benchmark it. Public storage belongs in a final { } block, while the measured prove function stays entirely in the proof context.
Prerequisites
Install Leo 4.4, Node.js 22 or newer, Rust, jq, and GNU time. On Ubuntu, the last two packages are available through apt:
sudo apt-get update
sudo apt-get install -y jq time
leo --version
node --version
rustc --version
Use a quiet machine. Pin RAYON_NUM_THREADS, close CPU-heavy applications, and keep the power mode unchanged between revisions. Linux users can add taskset if the machine has asymmetric cores or if other workloads share the host.
Create the project:
mkdir aleo-proving-bench
cd aleo-proving-bench
mkdir -p leo/src runner results artifacts
Step-by-step
1. Add the Leo workload
Create leo/program.json:
{
"program": "proving_bench.aleo",
"version": "0.1.0",
"description": "Fixed workload for reproducible proving benchmarks",
"license": "MIT",
"leo": "4.4.0",
"dependencies": null,
"dev_dependencies": null
}
Create leo/src/main.leo:
struct Round {
state: field,
domain: field,
index: u8,
}
fn mix(seed: field, domain: field) -> field {
let state: field = Poseidon2::hash_to_field(Round { state: seed, domain: domain, index: 0u8 });
for i: u8 in 0u8..64u8 {
state = Poseidon2::hash_to_field(Round { state: state, domain: domain, index: i + 1u8 });
}
return state;
}
program proving_bench.aleo {
@noupgrade
constructor() {}
storage published_runs: u64;
storage last_digest: field;
fn prove(seed: field, public domain: field) -> field {
return mix(seed, domain);
}
fn publish(public digest: field) -> Final {
return final {
let count: u64 = published_runs.unwrap_or(0u64);
published_runs = count + 1u64;
last_digest = digest;
};
}
}
mix lives outside the program because Leo 4.4 helper functions are top-level declarations. Every iteration depends on the previous hash, so the compiler cannot replace the loop with independent parallel work. The fixed bound also gives every revision the same circuit.
prove takes a private seed and a public domain separator. Its returned field is private by default. The function has no final block, network lookup, record scan, or mutable state, which makes it suitable for local proof generation.
publish demonstrates where storage belongs. It increments a public counter and stores a digest inside final { }. Keeping it out of the measured path avoids mixing proof time with finalization behavior.
The constructor is mandatory in Leo 4.4. @noupgrade also freezes the program after deployment, though deployment is not needed for this benchmark.
Build and run the workload:
cd leo
leo build
leo run prove 1field 7field
leo execute prove 1field 7field --print --json-output=build/execute.json
cd ..
leo run checks the local computation without generating a proof. leo execute is the slower smoke test that builds a proof-bearing execution transaction. Our measured runs use the SDK so one process can retain its prepared state between samples.
Find the compiled Aleo Instructions file:
find leo/build -type f -name 'proving_bench.aleo' -print
On a standard single-program build, the path is leo/build/proving_bench/proving_bench.aleo.
2. Add the benchmark runner
Create runner/package.json:
{
"name": "aleo-proving-bench-runner",
"private": true,
"type": "module",
"scripts": {
"bench": "tsx bench.ts",
"compare": "tsx compare.ts"
},
"dependencies": {
"@provablehq/sdk": "0.11.6"
},
"devDependencies": {
"tsx": "4.20.5",
"typescript": "5.9.2"
}
}
Create runner/bench.ts:
import fs from "node:fs";
import os from "node:os";
import { performance } from "node:perf_hooks";
import {
Account,
ProgramManager,
initThreadPool,
} from "@provablehq/sdk/mainnet.js";
const programPath = process.env.PROGRAM_PATH;
const revision = process.env.SNARKVM_REV;
const privateKey = process.env.PRIVATE_KEY;
const warmRuns = Number(process.env.WARM_RUNS ?? "10");
const blockHeight = Number(process.env.BLOCK_HEIGHT ?? "9000000");
const inputs = ["1field", "7field"];
if (!programPath || !revision || !privateKey) {
throw new Error("PROGRAM_PATH, SNARKVM_REV, and PRIVATE_KEY are required");
}
if (!Number.isInteger(warmRuns) || warmRuns < 1) {
throw new Error("WARM_RUNS must be a positive integer");
}
await initThreadPool();
const program = fs.readFileSync(programPath, "utf8");
const account = new Account({ privateKey });
function manager(): ProgramManager {
const value = new ProgramManager(undefined, undefined, undefined);
value.setAccount(account);
return value;
}
const setupManager = manager();
const [provingKey, verifyingKey] = await setupManager.synthesizeKeys(
program,
"prove",
inputs,
);
async function prove(
programManager: ProgramManager,
phase: "cold" | "warm",
sample: number,
): Promise<void> {
const started = performance.now();
const response = await programManager.run(
program,
"prove",
inputs,
true,
undefined,
undefined,
provingKey,
verifyingKey,
);
const elapsedMs = performance.now() - started;
const verified = programManager.verifyExecution(response, blockHeight);
const row = {
type: "sample",
revision,
phase,
sample,
wall_ms: Number(elapsedMs.toFixed(3)),
rss_bytes_after: process.memoryUsage().rss,
verified,
threads: process.env.RAYON_NUM_THREADS ?? "default",
cpu: os.cpus()?.model ?? "unknown",
platform: `${process.platform}-${process.arch}`,
node: process.version,
};
process.stdout.write(`${JSON.stringify(row)}\n`);
if (!verified) {
throw new Error(`proof verification failed during ${phase} sample ${sample}`);
}
}
const provingManager = manager();
await prove(provingManager, "cold", 0);
for (let sample = 1; sample <= warmRuns; sample += 1) {
await prove(provingManager, "warm", sample);
}
Key synthesis happens before the timer starts. The runner passes the resulting proving and verifying keys directly into every call, so a cache lookup or accidental resynthesis cannot masquerade as proving work.
The cold sample creates a fresh manager. Warm samples reuse it. Every response goes through verifyExecution; a false result stops the run immediately and the comparison script rejects any invalid sample left in a partial output file.
rss_bytes_after is a useful per-sample snapshot, but it is not peak memory. JavaScript timers cannot reliably sample RSS while a long WebAssembly call owns the thread. GNU time will record the process peak from outside Node.
3. Record wall time and peak RSS
Create runner/run-one.sh:
#!/usr/bin/env bash
set -euo pipefail
label=${1:?usage: run-one.sh LABEL SDK_TARBALL SNARKVM_REV}
tarball=${2:?usage: run-one.sh LABEL SDK_TARBALL SNARKVM_REV}
revision=${3:?usage: run-one.sh LABEL SDK_TARBALL SNARKVM_REV}
root=$(cd "$(dirname "$0")/.." && pwd)
out="$root/results/$label.jsonl"
time_file="$root/results/$label.time"
cd "$root/runner"
npm install --no-save "$tarball"
export PROGRAM_PATH="$root/leo/build/proving_bench/proving_bench.aleo"
export SNARKVM_REV="$revision"
export RAYON_NUM_THREADS=${RAYON_NUM_THREADS:-8}
export WARM_RUNS=${WARM_RUNS:-10}
/usr/bin/time -v npm run --silent bench > "$out" 2> "$time_file"
peak_kib=$(awk -F: '/Maximum resident set size/ {gsub(/^[[:space:]]+/, "", $2); print $2}' "$time_file")
jq -cn \
--arg revision "$revision" \
--arg label "$label" \
--argjson peak_kib "$peak_kib" \
'{type:"process", revision:$revision, label:$label, peak_rss_kib:$peak_kib}' \
>> "$out"
cat "$out"
Make it executable and install the runner:
chmod +x runner/run-one.sh
cd runner
npm install
cd ..
Each SDK tarball must be built from a clean checkout whose snarkVM dependency is pinned to the commit being tested. Do not relabel one prebuilt package with two commit hashes. Record the SDK commit, snarkVM commit, Rust version, lockfile checksum, and build flags beside the artifact.
For PR #3399, prepare one artifact from the parent commit and another from the merged commit. The SDK's Aleo WASM package contains the snarkVM proving code, so changing the revision requires rebuilding that package. A published npm package cannot be retargeted after installation.
Run both builds with the same host settings:
export PRIVATE_KEY='APrivateKey1...benchmark-only-key...'
export RAYON_NUM_THREADS=8
export WARM_RUNS=15
./runner/run-one.sh \
before \
"$PWD/artifacts/sdk-before.tgz" \
'<full-parent-commit-sha>'
./runner/run-one.sh \
after \
"$PWD/artifacts/sdk-after.tgz" \
'<full-merged-commit-sha>'
Use a dedicated local key. Never put a funded private key in benchmark scripts, shell history, or CI artifacts.
4. Add the regression gate
Create runner/compare.ts:
import fs from "node:fs";
const [beforePath, afterPath] = process.argv.slice(2);
const limit = Number(process.env.REGRESSION_LIMIT_PERCENT ?? "5");
if (!beforePath || !afterPath) {
throw new Error("usage: tsx compare.ts BEFORE.jsonl AFTER.jsonl");
}
type Sample = {
type: "sample";
phase: "cold" | "warm";
wall_ms: number;
verified: boolean;
};
type ProcessRow = {
type: "process";
peak_rss_kib: number;
};
function rows(path: string): Array<Sample | ProcessRow> {
return fs.readFileSync(path, "utf8")
.trim()
.split("\n")
.map((line) => JSON.parse(line));
}
function median(values: number[]): number {
const sorted = [...values].sort((a, b) => a - b);
const middle = Math.floor(sorted.length / 2);
return sorted.length % 2 === 0
? (sorted[middle - 1] + sorted[middle]) / 2
: sorted[middle];
}
function report(path: string) {
const data = rows(path);
const samples = data.filter((row): row is Sample => row.type === "sample");
const processRow = data.find((row): row is ProcessRow => row.type === "process");
if (samples.length === 0 || samples.some((row) => !row.verified) || !processRow) {
throw new Error(`invalid benchmark data in ${path}`);
}
return {
cold_ms: samples.find((row) => row.phase === "cold")!.wall_ms,
warm_median_ms: median(
samples.filter((row) => row.phase === "warm").map((row) => row.wall_ms),
),
peak_rss_kib: processRow.peak_rss_kib,
};
}
const before = report(beforePath);
const after = report(afterPath);
const timeChange = ((after.warm_median_ms / before.warm_median_ms) - 1) * 100;
const memoryChange = ((after.peak_rss_kib / before.peak_rss_kib) - 1) * 100;
console.log(JSON.stringify({ before, after, timeChange, memoryChange }, null, 2));
if (timeChange > limit || memoryChange > limit) {
process.exitCode = 1;
}
Run the gate:
cd runner
REGRESSION_LIMIT_PERCENT=5 npm run --silent compare -- \
../results/before.jsonl \
../results/after.jsonl
cd ..
A five percent threshold is a starting policy, not a law. On noisy developer machines, run several complete batches and compare their medians. CI runners need dedicated cores before a tighter threshold becomes credible.
5. Run the snarkVM microbenchmark
The end-to-end Leo result tells you whether users benefit. The algorithm benchmark tells you whether the edited function improved.
Use separate Git worktrees so build products cannot cross-contaminate revisions:
git clone [https://github.com/ProvableHQ/snarkVM.git](https://github.com/ProvableHQ/snarkVM.git) snarkvm
cd snarkvm
git worktree add ../snarkvm-before '<full-parent-commit-sha>'
git worktree add ../snarkvm-after '<full-merged-commit-sha>'
cd ..
for tree in snarkvm-before snarkvm-after; do
(
cd "$tree"
cargo bench -p snarkvm-algorithms -- --list \
| grep evaluate_all_lagrange_coefficients
RAYON_NUM_THREADS=8 cargo bench -p snarkvm-algorithms -- \
evaluate_all_lagrange_coefficients
)
done
Listing the benchmark first prevents a dangerously quiet failure where a renamed filter runs zero tests. Keep each worktree's target directory separate. Sharing compiled objects is fine for build speed in many projects, but here it weakens the audit trail around the tested revision.
Testing
Run the full local sequence:
cd leo
leo clean
leo build
leo run prove 1field 7field
leo test --prove
cd ..
export PRIVATE_KEY='APrivateKey1...benchmark-only-key...'
export RAYON_NUM_THREADS=8
export WARM_RUNS=15
./runner/run-one.sh before "$PWD/artifacts/sdk-before.tgz" '<parent-sha>'
./runner/run-one.sh after "$PWD/artifacts/sdk-after.tgz" '<merged-sha>'
cd runner
npm run --silent compare -- ../results/before.jsonl ../results/after.jsonl
Check four things before accepting a speedup:
- Every sample has
"verified": true. - Both result files report the intended full commit SHA and thread count.
- Cold time, warm median time, and peak RSS are compared separately.
- The isolated Lagrange benchmark moves in the same direction as the Leo workload.
Do not average cold and warm samples together. I would also reject a result produced after changing inputs, compiler versions, constraint counts, or worker counts. Faster invalid proofs deserve a bug report, not a victory lap.
Random proof bytes are expected. Equal inputs do not imply byte-for-byte equal proofs because proof generation uses randomness. Verification success, circuit identity, fixed inputs, and measured resource use are the invariants we care about.
What's next
Add a second workload with a different circuit shape, perhaps scalar arithmetic or signature verification. One hash-heavy program cannot represent every Aleo application.
The earlier reusable proving context tutorial pairs well with this project. Prepared contexts remove repeated program preparation from batch jobs. Apply the same discipline here: measure setup on purpose, measure proving on purpose, and never blend the two numbers.
For CI, archive the JSONL files with /proc/cpuinfo, the SDK tarball checksum, rustc -Vv, and the full snarkVM SHA. Run on dedicated hardware when a few percent matters. Shared cloud runners are fine for catching a 40 percent slowdown and poor at judging a 2 percent win.
PR #3399 changes work inside a hot polynomial routine. The benchmark should therefore show its clearest signal in evaluate_all_lagrange_coefficients, with a smaller effect in the complete Leo proof. If the end-to-end result gets dramatically faster while the microbenchmark barely moves, investigate setup, caches, or artifact mismatches before believing it.