Introduction
Historical Aleo ingestion used to mean downloading complete blocks, even when an indexer wanted only confirmed transactions. Headers, signatures, ratifications, solutions, and other block material crossed the wire and then went straight into the bin.
snarkOS PR #4455 added blocks/transactions, a compact range route that returns one transaction array per requested block. The range uses an inclusive start and exclusive end, matching the other bulk block routes. Asking for start=800&end=850 covers heights 800 through 849.
The first implementation capped a request at 50 blocks. PR #4467 proposes raising that ceiling to 160 because transaction-only responses are smaller than complete block responses. Good change. Hard-coding 160 into a client would still be a mistake.
Node versions move at different speeds. Public gateways may lag self-hosted nodes, and reverse proxies can impose smaller limits of their own. Our client will probe the endpoint, find the largest accepted range under a local ceiling, and remember the result.
What we're building
The backfiller will:
- Read
GET /blocks/transactions?start=H&end=Hwithout downloading complete blocks. - Discover a safe batch size with binary search instead of assuming 50 or 160.
- Compare the first and last transaction arrays against the single-block route.
- Retry rate limits, timeouts, and temporary server failures with jittered backoff.
- Write each batch and its manifest through atomic renames.
- Resume at the first uncommitted height after a crash.
Each batch becomes an independent JSON file. The manifest stays small and records only the next height, negotiated range, segment count, and most recent segment hash.
I prefer immutable batch files over one giant JSONL append log. An interrupted append creates an awkward question: did the final line reach disk? Atomic batch replacement has a cleaner answer. Either the named segment exists in full, or the manifest does not refer to it.
We will also build a small Leo program that can publish selected checkpoint digests. Do this sparingly. Anchoring every batch would turn an indexer into a very committed fee-burning machine.
Prerequisites
You need Node.js 20 or newer, npm, Leo 4.4.0, and access to a snarkOS REST endpoint containing the new range route.
A self-hosted v2 endpoint might look like:
http://127.0.0.1:3030/v2/testnet
A gateway may use another prefix. Pass the complete network root through SNARKOS_URL; the script appends only route paths.
Create the project:
mkdir adaptive-aleo-backfiller
cd adaptive-aleo-backfiller
mkdir -p src data anchor/src
Step-by-step
1. Add the TypeScript package
Create package.json:
{
"name": "adaptive-aleo-backfiller",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"backfill": "tsx src/backfill.ts",
"typecheck": "tsc --noEmit"
},
"devDependencies": {
"@types/node": "^24.0.0",
"tsx": "^4.20.0",
"typescript": "^5.9.0"
}
}
Create tsconfig.json:
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"noUncheckedIndexedAccess": true,
"skipLibCheck": true,
"types": ["node"]
},
"include": ["src/**/*.ts"]
}
Keep generated data and Leo build output out of Git:
node_modules/
data/
anchor/build/
anchor/.env
Install the toolchain:
npm install
No runtime library is needed. Node already supplies fetch, SHA-256, filesystem primitives, and abortable requests.
2. Build the backfiller
Create src/backfill.ts:
import { createHash } from "node:crypto";
import { mkdir, open, readFile, rename } from "node:fs/promises";
import path from "node:path";
const root = required("SNARKOS_URL").replace(/\/$/, "");
const requestedStart = integer("START_HEIGHT");
const targetEnd = integer("END_HEIGHT");
const probeCeiling = optionalInteger("PROBE_CEILING", 256);
const timeoutMs = optionalInteger("REQUEST_TIMEOUT_MS", 30_000);
const renegotiate = process.env.RENEGOTIATE === "1";
const dataDir = path.resolve(process.env.DATA_DIR ?? "data");
const manifestPath = path.join(dataDir, "manifest.json");
type JsonObject = Record<string, unknown>;
type Transaction = JsonObject;
type BlockTransactions = Transaction[];
type Segment = {
file: string;
start: number;
end: number;
sha256: string;
anchorField: string;
};
type Manifest = {
version: 1;
source: string;
startHeight: number;
nextHeight: number;
endHeight: number;
batchSize: number;
segmentCount: number;
lastSegment: Segment | null;
updatedAt: string;
};
class HttpError extends Error {
constructor(
readonly status: number,
readonly body: string,
readonly url: string,
) {
super(`HTTP ${status} from ${url}: ${body.slice(0, 300)}`);
}
}
function required(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`${name} is required`);
return value;
}
function integer(name: string): number {
const value = Number(required(name));
if (!Number.isSafeInteger(value) || value < 0) {
throw new Error(`${name} must be a non-negative safe integer`);
}
return value;
}
function optionalInteger(name: string, fallback: number): number {
const raw = process.env[name];
if (raw === undefined) return fallback;
const value = Number(raw);
if (!Number.isSafeInteger(value) || value < 1) {
throw new Error(`${name} must be a positive safe integer`);
}
return value;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function isRangeError(error: unknown): boolean {
if (!(error instanceof HttpError)) return false;
if ([400, 413, 422].includes(error.status)) return true;
return error.status === 500 &&
/(range|maximum|max.*block|more than).*\d/i.test(error.body);
}
function isTransient(error: unknown): boolean {
if (error instanceof HttpError) {
return error.status === 408 || error.status === 429 || error.status >= 500;
}
return error instanceof TypeError ||
(error instanceof Error && error.name === "TimeoutError");
}
async function fetchJsonOnce(url: string): Promise<unknown> {
const response = await fetch(url, {
headers: { accept: "application/json" },
signal: AbortSignal.timeout(timeoutMs),
});
const body = await response.text();
if (!response.ok) throw new HttpError(response.status, body, url);
try {
return JSON.parse(body) as unknown;
} catch {
throw new Error(`Invalid JSON from ${url}`);
}
}
async function fetchJson(url: string): Promise<unknown> {
for (let attempt = 0; attempt < 6; attempt++) {
try {
return await fetchJsonOnce(url);
} catch (error) {
if (isRangeError(error) || !isTransient(error) || attempt === 5) throw error;
const base = 250 * 2 ** attempt;
await sleep(base + Math.floor(Math.random() * base));
}
}
throw new Error("unreachable retry state");
}
function parseBlockArrays(value: unknown, span: number): BlockTransactions[] {
if (!Array.isArray(value)) throw new Error("Range response is not an array");
if (value.length !== span) {
throw new Error(`Expected ${span} block arrays, received ${value.length}`);
}
for (const [index, block] of value.entries()) {
if (!Array.isArray(block)) {
throw new Error(`Range response item ${index} is not a transaction array`);
}
}
return value as BlockTransactions[];
}
async function fetchRange(start: number, end: number): Promise<BlockTransactions[]> {
const url = new URL(`${root}/blocks/transactions`);
url.searchParams.set("start", String(start));
url.searchParams.set("end", String(end));
return parseBlockArrays(await fetchJson(url.toString()), end - start);
}
async function fetchOne(height: number): Promise<BlockTransactions> {
const value = await fetchJson(`${root}/block/${height}/transactions`);
if (!Array.isArray(value)) {
throw new Error(`Single-block response for ${height} is not an array`);
}
return value as BlockTransactions;
}
function transactionId(value: Transaction): string | undefined {
const nested = value.transaction;
if (nested && typeof nested === "object" && !Array.isArray(nested)) {
const id = (nested as JsonObject).id;
if (typeof id === "string") return id;
}
for (const key of ["id", "transaction_id"]) {
if (typeof value[key] === "string") return value[key] as string;
}
return undefined;
}
function transactionSignature(items: BlockTransactions): string {
const ids = items.map(transactionId);
if (ids.every((id): id is string => id !== undefined)) return ids.join("\n");
return JSON.stringify(items);
}
async function validateBoundaries(
start: number,
end: number,
blocks: BlockTransactions[],
): Promise<void> {
const heights = start === end - 1 ? [start] : [start, end - 1];
for (const height of heights) {
const expected = await fetchOne(height);
const actual = blocks[height - start];
if (!actual || transactionSignature(actual) !== transactionSignature(expected)) {
throw new Error(`Boundary mismatch at block ${height}`);
}
}
}
async function negotiateRange(
start: number,
end: number,
): Promise<{ size: number; blocks: BlockTransactions[] }> {
let low = 1;
let high = Math.min(probeCeiling, end - start);
let best: { size: number; blocks: BlockTransactions[] } | null = null;
while (low <= high) {
const size = Math.floor((low + high) / 2);
try {
const blocks = await fetchRange(start, start + size);
best = { size, blocks };
low = size + 1;
} catch (error) {
if (!isRangeError(error)) throw error;
high = size - 1;
}
}
if (!best) throw new Error("The endpoint rejected a one-block transaction range");
return best;
}
function digest(value: string): string {
return createHash("sha256").update(value).digest("hex");
}
function digestField(sha256: string): string {
const first31Bytes = sha256.slice(0, 62);
return `${BigInt(`0x${first31Bytes}`).toString(10)}field`;
}
function segmentName(start: number, end: number): string {
return `${String(start).padStart(12, "0")}-${String(end).padStart(12, "0")}.json`;
}
async function atomicWrite(file: string, value: string): Promise<void> {
const temporary = `${file}.${process.pid}.${Date.now()}.tmp`;
const handle = await open(temporary, "wx");
try {
await handle.writeFile(value, "utf8");
await handle.sync();
} finally {
await handle.close();
}
await rename(temporary, file);
}
async function loadManifest(): Promise<Manifest | null> {
try {
return JSON.parse(await readFile(manifestPath, "utf8")) as Manifest;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
throw error;
}
}
async function main(): Promise<void> {
if (targetEnd <= requestedStart) {
throw new Error("END_HEIGHT must be greater than START_HEIGHT");
}
await mkdir(dataDir, { recursive: true });
let manifest = await loadManifest();
if (manifest && manifest.source !== root) {
throw new Error("Manifest source does not match SNARKOS_URL");
}
if (manifest && manifest.startHeight !== requestedStart) {
throw new Error("START_HEIGHT does not match the existing manifest");
}
if (manifest && targetEnd < manifest.nextHeight) {
throw new Error("END_HEIGHT is behind the committed checkpoint");
}
let cursor = manifest?.nextHeight ?? requestedStart;
let batchSize = manifest?.batchSize ?? 0;
let reusable: BlockTransactions[] | null = null;
if (cursor < targetEnd && (!manifest || renegotiate)) {
const negotiated = await negotiateRange(cursor, targetEnd);
batchSize = negotiated.size;
reusable = negotiated.blocks;
console.log(`Negotiated ${batchSize} blocks per request`);
}
manifest ??= {
version: 1,
source: root,
startHeight: requestedStart,
nextHeight: requestedStart,
endHeight: targetEnd,
batchSize,
segmentCount: 0,
lastSegment: null,
updatedAt: new Date().toISOString(),
};
while (cursor < targetEnd) {
const end = Math.min(cursor + batchSize, targetEnd);
const blocks = reusable && reusable.length === end - cursor
? reusable
: await fetchRange(cursor, end);
reusable = null;
await validateBoundaries(cursor, end, blocks);
const core = {
version: 1,
source: root,
start: cursor,
end,
fetchedAt: new Date().toISOString(),
blocks,
};
const serializedCore = JSON.stringify(core);
const sha256 = digest(serializedCore);
const anchorField = digestField(sha256);
const file = segmentName(cursor, end);
const segmentPath = path.join(dataDir, file);
await atomicWrite(
segmentPath,
`${JSON.stringify({ ...core, sha256, anchorField }, null, 2)}\n`,
);
const segment: Segment = { file, start: cursor, end, sha256, anchorField };
manifest = {
...manifest,
nextHeight: end,
endHeight: targetEnd,
batchSize,
segmentCount: manifest.segmentCount + 1,
lastSegment: segment,
updatedAt: new Date().toISOString(),
};
await atomicWrite(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`);
console.log(`Committed [${cursor}, ${end}) as ${file}`);
cursor = end;
}
}
main().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
The binary search is intentionally bounded by PROBE_CEILING. A client has no reason to test absurd ranges merely because a server forgot to reject them. With the default ceiling of 256, a 50-block node settles on 50, while a node accepting the proposed limit settles on 160.
Range failures need careful classification. A v2 snarkOS route can report bad ranges as HTTP 400. Older prefixes may rewrite the same handler error to HTTP 500, so the client also checks the response text. An unrelated 500 remains retryable and never becomes evidence about the range limit.
Boundary checks protect against bad proxy routing and response-shape mistakes. The bulk result at height start must match /block/{start}/transactions; the same comparison runs at end - 1. Empty blocks still occupy an outer array slot, which is why the script rejects a response whose outer length differs from end - start.
After writing a segment, the script calls fsync, renames the temporary file, and only then advances the manifest. A crash between those renames can leave an unreferenced segment file. The next run safely replaces that filename before committing it.
3. Run a backfill
Type-check first:
npm run typecheck
Backfill heights 1,000 through 4,999:
SNARKOS_URL=http://127.0.0.1:3030/v2/testnet \
START_HEIGHT=1000 \
END_HEIGHT=5000 \
PROBE_CEILING=256 \
npm run backfill
END_HEIGHT is exclusive. To include a chain tip reported as height 5,000, pass END_HEIGHT=5001.
Restart with the same command after an interruption. The script reads data/manifest.json and resumes from nextHeight. To test for a larger server limit after upgrading a node, add:
RENEGOTIATE=1 npm run backfill
Keep the other environment variables set. The newly accepted size is saved only after negotiation succeeds.
4. Add the Leo anchor
The local manifest is enough for ordinary recovery. A public checkpoint can help when independent operators need to agree on one exported snapshot.
Create anchor/program.json:
{
"program": "backfill_anchor.aleo",
"version": "0.1.0",
"description": "Publishes selected Aleo backfill checkpoint digests",
"license": "MIT",
"dependencies": null,
"dev_dependencies": null
}
Create anchor/src/main.leo:
const ADMIN: address = aleo1rhgdu77hgyqd3xjj8ucu3jj9r2krwz6mnzyd80gncr5fxcwlh5rsvzp9px;
program backfill_anchor.aleo {
storage last_height: u32;
storage last_digest: field;
fn publish(public end_height: u32, public digest: field) -> Final {
let signer: address = std::ctx::signer();
assert_eq(signer, ADMIN);
return final {
let current: u32 = last_height.unwrap_or(0u32);
assert(end_height > current);
last_height = end_height;
last_digest = digest;
};
}
@noupgrade
constructor() {}
}
Replace ADMIN before a real deployment. The address above is the published local devnode account, so it provides no security on a public network.
publish captures the original transaction signer in the proof half and checks it against ADMIN. Its final block reads the current singleton with an explicit default, rejects checkpoint regression, and writes the new exclusive end height with its digest field.
The TypeScript process takes the first 31 bytes of SHA-256 and converts them to a Leo field literal. Truncating to 248 bits guarantees that the integer fits in the field while retaining ample collision resistance for checkpoint identification. The full 256-bit hash remains in the segment and manifest.
The constructor uses @noupgrade, so deployed code cannot change later. That is my preferred policy for a tiny anchor whose value depends on predictable rules.
Build and run the proof half:
cd anchor
leo build
leo run publish 5000u32 123field
leo run checks the circuit portion but does not execute the final block. Exercise storage on a local devnode:
leo devnode start
leo deploy --broadcast --devnet
leo execute publish 5000u32 123field --broadcast --devnet
For a real checkpoint, copy anchorField from data/manifest.json and use lastSegment.end as the u32 height. Anchor weekly exports or named releases, not every fetched range.
Testing
Start with a deliberately oversized probe ceiling:
rm -rf data
SNARKOS_URL=http://127.0.0.1:3030/v2/testnet \
START_HEIGHT=1000 \
END_HEIGHT=1400 \
PROBE_CEILING=256 \
npm run backfill
Confirm that manifest.json records the accepted size rather than 256. A node with the original cap should record 50. A node carrying the proposed PR #4467 change should record 160.
Now interrupt the process after at least one Committed line, then rerun the same command. Previously committed segment files should remain untouched until the cursor reaches new work, and nextHeight should continue moving forward without duplicate manifest counts.
Test the boundary guard by placing a proxy in front of the node and making its single-block route point at another network. The first batch should fail with Boundary mismatch before either the segment or manifest becomes committed.
Finally, provoke a transient failure by returning HTTP 429 for the first few requests. The delay should grow on each attempt and include random jitter. A persistent malformed response should fail immediately; retrying bad JSON six times only makes a broken server six times slower to diagnose.
What's next
The next useful addition is a sink interface. Feed each committed segment into PostgreSQL, ClickHouse, or an object store, then advance the same manifest only after that sink commits. Preserve the write ordering used here: data first, checkpoint second.
For program-specific indexing, connect the transaction decoder to the checkpointed Aleo program activity indexer. Its deduplication model fits naturally after this backfiller's range fetch.
Operators with several nodes can borrow the comparison strategy from the multi-node Aleo chain consistency monitor. Negotiate a range per node rather than forcing the smallest node's setting on every source, then compare boundary transaction IDs before accepting a segment.
One architectural rule should survive all those extensions: server limits are observations, not constants. Probe them, validate the returned boundaries, and save what worked.