Introduction
Trusting one API endpoint is convenient. It is also a poor verification strategy for an indexer, explorer, bridge watcher, or anything else that makes decisions from Aleo chain data.
A node can lag. A reverse proxy can cache stale responses. An operator can restore the wrong database snapshot. None of those failures require broken consensus, yet each can feed your service a believable and internally consistent view of the wrong tip.
snarkOS PR 4447 added bulk projections for block hashes, headers, and state roots. The routes return narrow slices of block data without the transactions, solutions, ratifications, and other material carried by complete blocks. PR 4448 then exercised the real REST handlers, including range failures and version-specific error handling.
That pairing matters. Cheap projections make continuous comparison practical. Route-level tests make the behavior less mysterious.
We will build a TypeScript service that polls two or more independently operated nodes, verifies each node's state-root chain, compares their projections, reports the first divergent height, and saves a compact checkpoint with an atomic rename. We will also add a small Leo program for publishing selected checkpoint digests on-chain.
I would not call the Leo program from every polling cycle. That would turn an observability service into a transaction factory. Local checkpoints should be frequent; public anchors should be sparse and deliberate.
If you built the checkpointed Aleo program activity indexer, the persistence model will feel familiar. The difference is what we trust. That indexer tracks application activity. Here, independent nodes must agree before progress becomes durable.
Tags: project-of-the-week, snarkos, rest-api, typescript, indexing, observability
What we're building
The monitor uses these snarkOS range routes beneath a versioned network base URL:
GET /blocks/hashes?start=H&end=H
GET /blocks/headers?start=H&end=H
GET /blocks/stateRoots?start=H&end=H
For a node configured as http://127.0.0.1:3030/v2/testnet, the first path becomes:
http://127.0.0.1:3030/v2/testnet/blocks/hashes?start=120000&end=120049
Ranges are inclusive. snarkOS rejects a range whose end - start exceeds 50, so the service requests 50 returned heights at a time by setting end = start + 49. Keeping that arithmetic explicit avoids a lovely little fencepost incident at 3 a.m.
The monitor checks four properties:
- Every projection has the expected number of entries.
- Every header reports the height implied by its array position.
- A header's
previous_state_rootequals the preceding block's projected state root. - All nodes return the same block hash, state root, and canonicalized header at each height.
State-root continuity deserves emphasis. Aleo's block header contains previous_state_root, while the block hash's predecessor relationship lives outside the projected header. Comparing header[h].previous_state_root with stateRoot[h - 1] gives us a cheap continuity check without downloading full blocks.
The first disagreement stops the scan. We do not vote and move on. Two matching providers behind the same upstream are not independent, and a majority is meaningless when the deployment topology is unknown.
PR 4448 also documents an API-version wrinkle: a missing height is exposed as HTTP 404 on /v2. The default and /v1 route sets retain their older error mapping. Use /v2 for this service so a missing block can be separated from an invalid range or a server fault.
Prerequisites
You need:
- Node.js 22 or newer.
- TypeScript 5.7 or newer.
- At least two snarkOS REST endpoints operated through independent infrastructure.
- Leo 4.4.0 for the optional checkpoint anchor.
Do not compare two hostnames that terminate at the same gateway and call that redundancy. Use separate node operators, or run one snarkOS node yourself and compare it with an external provider.
Create the service:
mkdir aleo-chain-monitor
cd aleo-chain-monitor
npm init -y
npm install dotenv
npm install --save-dev typescript tsx @types/node
mkdir -p src data
Replace package.json with:
{
"name": "aleo-chain-consistency-monitor",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"build": "tsc -p tsconfig.json",
"start": "tsx src/index.ts",
"check": "tsc -p tsconfig.json --noEmit"
},
"dependencies": {
"dotenv": "^16.4.7"
},
"devDependencies": {
"@types/node": "^22.10.2",
"tsx": "^4.19.2",
"typescript": "^5.7.2"
}
}
Add tsconfig.json:
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"outDir": "dist",
"skipLibCheck": true
},
"include": ["src/**/*.ts"]
}
Create .env with your actual endpoints:
NETWORK=testnet
NODES=local=http://127.0.0.1:3030/v2/testnet,remote=[https://node.example.net/v2/testnet](https://node.example.net/v2/testnet)
START_HEIGHT=0
POLL_MS=15000
CHUNK_SIZE=50
CHECKPOINT_FILE=data/checkpoint.json
ONCE=false
Each node URL already includes the API version and network. That choice keeps route construction boring, which is exactly what URL construction should be.
Step-by-step
1. Add the monitor
Create src/index.ts:
import "dotenv/config";
import { createHash } from "node:crypto";
import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
import { dirname } from "node:path";
interface NodeConfig {
name: string;
baseUrl: string;
}
interface AleoHeader {
previous_state_root: string;
metadata: {
height: number | string;
};
[key: string]: unknown;
}
interface ProjectionBatch {
node: string;
start: number;
end: number;
hashes: string[];
headers: AleoHeader[];
roots: string[];
headerDigests: string[];
}
interface Checkpoint {
version: 1;
network: string;
height: number;
blockHash: string;
stateRoot: string;
headerDigest: string;
checkpointDigest: string;
savedAt: string;
}
const NETWORK = process.env.NETWORK ?? "testnet";
const START_HEIGHT = parseInteger(process.env.START_HEIGHT ?? "0", "START_HEIGHT");
const POLL_MS = parseInteger(process.env.POLL_MS ?? "15000", "POLL_MS");
const CHUNK_SIZE = parseInteger(process.env.CHUNK_SIZE ?? "50", "CHUNK_SIZE");
const CHECKPOINT_FILE = process.env.CHECKPOINT_FILE ?? "data/checkpoint.json";
const ONCE = process.env.ONCE === "true";
if (CHUNK_SIZE < 1 || CHUNK_SIZE > 50) {
throw new Error("CHUNK_SIZE must be between 1 and 50");
}
const nodes = parseNodes(process.env.NODES ?? "");
if (nodes.length < 2) {
throw new Error("NODES must contain at least two named endpoints");
}
function parseInteger(value: string, name: string): number {
const parsed = Number(value);
if (!Number.isSafeInteger(parsed) || parsed < 0) {
throw new Error(`${name} must be a non-negative integer`);
}
return parsed;
}
function parseNodes(value: string): NodeConfig[] {
return value.split(",").filter(Boolean).map((entry) => {
const separator = entry.indexOf("=");
if (separator < 1) throw new Error(`Invalid NODES entry: ${entry}`);
return {
name: entry.slice(0, separator),
baseUrl: entry.slice(separator + 1).replace(/\/$/, "")
};
});
}
async function fetchJson<T>(url: string): Promise<T> {
const response = await fetch(url, {
headers: { accept: "application/json" },
signal: AbortSignal.timeout(10_000)
});
if (!response.ok) {
const body = await response.text();
throw new Error(`${response.status} ${response.statusText} from ${url}: ${body}`);
}
return JSON.parse(await response.text()) as T;
}
async function fetchRange<T>(
node: NodeConfig,
route: string,
start: number,
end: number
): Promise<T[]> {
const url = new URL(`${node.baseUrl}/${route}`);
url.searchParams.set("start", String(start));
url.searchParams.set("end", String(end));
return fetchJson<T[]>(url.toString());
}
async function latestHeight(node: NodeConfig): Promise<number> {
const value = await fetchJson<number | string>(`${node.baseUrl}/latest/height`);
return parseInteger(String(value), `latest height from ${node.name}`);
}
function canonicalJson(value: unknown): string {
if (Array.isArray(value)) {
return `[${value.map(canonicalJson).join(",")}]`;
}
if (value !== null && typeof value === "object") {
const object = value as Record<string, unknown>;
const entries = Object.keys(object).sort().map(
(key) => `${JSON.stringify(key)}:${canonicalJson(object[key])}`
);
return `{${entries.join(",")}}`;
}
return JSON.stringify(value);
}
function sha256(value: string): string {
return createHash("sha256").update(value).digest("hex");
}
function headerHeight(header: AleoHeader): number {
const raw = String(header.metadata.height).replace(/u\d+$/, "");
return parseInteger(raw, "header height");
}
async function loadBatch(
node: NodeConfig,
start: number,
end: number
): Promise<ProjectionBatch> {
const [hashes, headers, roots] = await Promise.all([
fetchRange<string>(node, "blocks/hashes", start, end),
fetchRange<AleoHeader>(node, "blocks/headers", start, end),
fetchRange<string>(node, "blocks/stateRoots", start, end)
]);
const expected = end - start + 1;
for (const [name, values] of Object.entries({ hashes, headers, roots })) {
if (values.length !== expected) {
throw new Error(
`${node.name} returned ${values.length} ${name} entries for ${expected} heights`
);
}
}
let priorRoot: string | undefined;
if (start > 0) {
const prior = await fetchRange<string>(node, "blocks/stateRoots", start - 1, start - 1);
if (prior.length !== 1) throw new Error(`${node.name} did not return the prior state root`);
priorRoot = prior;
}
for (let index = 0; index < headers.length; index += 1) {
const height = start + index;
const reportedHeight = headerHeight(headers[index]);
if (reportedHeight !== height) {
throw new Error(`${node.name} returned header ${reportedHeight} at position ${height}`);
}
const expectedPreviousRoot = index === 0 ? priorRoot : roots[index - 1];
if (
expectedPreviousRoot !== undefined &&
headers[index].previous_state_root !== expectedPreviousRoot
) {
throw new Error(
`${node.name} has broken state-root continuity at height ${height}`
);
}
}
return {
node: node.name,
start,
end,
hashes,
headers,
roots,
headerDigests: headers.map((header) => sha256(canonicalJson(header)))
};
}
function firstDivergence(batches: ProjectionBatch[]): number | undefined {
const baseline = batches;
for (let index = 0; index < baseline.hashes.length; index += 1) {
for (const candidate of batches.slice(1)) {
if (
candidate.hashes[index] !== baseline.hashes[index] ||
candidate.roots[index] !== baseline.roots[index] ||
candidate.headerDigests[index] !== baseline.headerDigests[index]
) {
return baseline.start + index;
}
}
}
return undefined;
}
function describeDivergence(height: number, batches: ProjectionBatch[]): string {
const index = height - batches.start;
const lines = [`Divergence at Aleo height ${height}`];
for (const batch of batches) {
lines.push(
`${batch.node}: hash=${batch.hashes[index]} root=${batch.roots[index]} header=${batch.headerDigests[index]}`
);
}
return lines.join("\n");
}
async function readCheckpoint(): Promise<Checkpoint | undefined> {
try {
return JSON.parse(await readFile(CHECKPOINT_FILE, "utf8")) as Checkpoint;
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
if (code === "ENOENT") return undefined;
throw error;
}
}
async function saveCheckpoint(checkpoint: Checkpoint): Promise<void> {
await mkdir(dirname(CHECKPOINT_FILE), { recursive: true });
const temporary = `${CHECKPOINT_FILE}.${process.pid}.tmp`;
await writeFile(temporary, `${JSON.stringify(checkpoint)}\n`, "utf8");
await rename(temporary, CHECKPOINT_FILE);
}
function checkpointFrom(batch: ProjectionBatch): Checkpoint {
const index = batch.hashes.length - 1;
const height = batch.end;
const blockHash = batch.hashes[index];
const stateRoot = batch.roots[index];
const headerDigest = batch.headerDigests[index];
const checkpointDigest = sha256(
`${NETWORK}:${height}:${blockHash}:${stateRoot}:${headerDigest}`
);
return {
version: 1,
network: NETWORK,
height,
blockHash,
stateRoot,
headerDigest,
checkpointDigest,
savedAt: new Date().toISOString()
};
}
async function verifyCheckpoint(checkpoint: Checkpoint): Promise<void> {
if (checkpoint.network !== NETWORK) {
throw new Error(`Checkpoint is for ${checkpoint.network}, not ${NETWORK}`);
}
const batches = await Promise.all(
nodes.map((node) => loadBatch(node, checkpoint.height, checkpoint.height))
);
const divergence = firstDivergence(batches);
if (
divergence !== undefined ||
batches.hashes !== checkpoint.blockHash ||
batches.roots !== checkpoint.stateRoot ||
batches.headerDigests !== checkpoint.headerDigest
) {
throw new Error(`Saved checkpoint no longer matches height ${checkpoint.height}`);
}
}
async function scanOnce(): Promise<void> {
const checkpoint = await readCheckpoint();
if (checkpoint) await verifyCheckpoint(checkpoint);
const tips = await Promise.all(nodes.map(latestHeight));
const commonTip = Math.min(...tips);
let cursor = checkpoint ? checkpoint.height + 1 : START_HEIGHT;
console.log(
`tips ${nodes.map((node, index) => `${node.name}=${tips[index]}`).join(" ")} common=${commonTip}`
);
while (cursor <= commonTip) {
const end = Math.min(cursor + CHUNK_SIZE - 1, commonTip);
const batches = await Promise.all(nodes.map((node) => loadBatch(node, cursor, end)));
const divergence = firstDivergence(batches);
if (divergence !== undefined) {
throw new Error(describeDivergence(divergence, batches));
}
const nextCheckpoint = checkpointFrom(batches);
await saveCheckpoint(nextCheckpoint);
console.log(
`agreed ${cursor}-${end} hash=${nextCheckpoint.blockHash} root=${nextCheckpoint.stateRoot}`
);
cursor = end + 1;
}
}
async function main(): Promise<void> {
do {
await scanOnce();
if (!ONCE) await new Promise((resolve) => setTimeout(resolve, POLL_MS));
} while (!ONCE);
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exit(1);
});
2. Understand the failure boundary
loadBatch treats every node as hostile input. Array lengths, header heights, and state-root continuity are checked before cross-node comparison begins.
Canonicalizing the header before hashing avoids false alarms caused by JSON object key order. Arrays keep their order because order can carry meaning. The SHA-256 digest is only a compact local fingerprint; it is not an Aleo block hash and must never be presented as one.
firstDivergence walks heights from low to high. Bulk requests already bound the search to 50 entries, so a binary search would add network round trips while saving a trivial local loop. I prefer the loop.
Checkpoint writes use a temporary file followed by rename. A crash can leave a temporary file behind, but it cannot leave half a JSON document at the configured checkpoint path.
3. Add a Leo checkpoint anchor
Local JSON is enough for restart safety. A public digest is useful when separate organizations want an auditable statement that an operator observed a given checkpoint.
Create the project:
leo new chain_monitor
cd chain_monitor
Replace src/main.leo with:
const ADMIN: address = aleo1qnr4dkkvkgfqph0vzc3y6z2eu975wnpz2925ntjccd5cfqxtyu8s7pyjh9;
program chain_monitor.aleo {
@noupgrade
constructor() {}
storage latest_height: u32;
mapping checkpoints: u32 => field;
fn commit_checkpoint(public height: u32, public digest: field) -> Final {
let signer: address = std::ctx::signer();
assert_eq(signer, ADMIN);
return final {
let previous: u32 = latest_height.unwrap_or(0u32);
assert(height > previous);
checkpoints.set(height, digest);
latest_height = height;
};
}
}
Replace program.json with the full manifest:
{
"program": "chain_monitor.aleo",
"version": "0.1.0",
"description": "Publishes sparse checkpoints from an Aleo chain consistency monitor",
"license": "MIT",
"leo": "4.4.0",
"dependencies": null,
"dev_dependencies": null
}
Build and run it:
leo build
leo run commit_checkpoint 120000u32 123456789field
leo run executes the proof half and returns a Final handle. It does not persist latest_height or checkpoints. Use leo execute against a devnet or broadcast network transaction when you want finalization to run:
leo execute commit_checkpoint 120000u32 123456789field --broadcast --devnet
The function captures std::ctx::signer() before entering final, since signer context is unavailable inside finalization. The storage singleton records the newest published height. The mapping retains the digest at each published height.
Replace ADMIN before deployment. A @noupgrade constructor makes that address permanent, which is desirable only after you have checked it twice.
Do not squeeze the monitor's hexadecimal SHA-256 string into a Leo field with an undocumented cast. Produce the public field through a specified conversion or an Aleo-native hash routine, record that scheme alongside the service version, and keep it stable. A checkpoint nobody can reproduce is decorative data.
Testing
Compile the TypeScript first:
cd aleo-chain-monitor
npm run check
npm run build
For a harmless smoke test, point both names at the same node and run one pass:
NODES=copy-a=http://127.0.0.1:3030/v2/testnet,copy-b=http://127.0.0.1:3030/v2/testnet \
ONCE=true \
npm start
You should see the reported tips followed by agreed ranges. Inspect the checkpoint:
cat data/checkpoint.json
A saved file looks like this:
{"version":1,"network":"testnet","height":120049,"blockHash":"ab1...","stateRoot":"sr1...","headerDigest":"8ca1...","checkpointDigest":"d971...","savedAt":"2026-09-14T07:00:00.000Z"}
Now use genuinely independent endpoints. Run once, stop the process, then run it again. The second run re-fetches the saved height from every node before advancing. Changing blockHash, stateRoot, or headerDigest in the JSON file should make startup fail.
To test divergence reporting without waiting for a real incident, place a tiny HTTP proxy in front of one development node and alter one hash in a copied response. The monitor should print the first affected height plus each node's hash, root, and header digest. It must not save a checkpoint for the failed range.
Exercise the route errors directly too:
curl -i "http://127.0.0.1:3030/v2/testnet/blocks/hashes?start=100&end=99"
curl -i "http://127.0.0.1:3030/v2/testnet/blocks/hashes?start=100&end=151"
curl -i "http://127.0.0.1:3030/v2/testnet/blocks/hashes?start=4294967295&end=4294967295"
The reversed and oversized ranges should be rejected. On /v2, a well-formed range containing a missing height should surface as 404. Alert on that separately from disagreement: a lagging node is different from two nodes returning conflicting data.
What's next
Run the monitor near each data consumer rather than behind one shared gateway. Export counters for the common tip, per-node lag, request latency, last agreed height, checkpoint age, and divergence height. Six metrics are enough to tell an operator whether the problem is delay, failure, or conflicting chain data.
A production deployment also needs reorganization policy. The sample refuses to move past a changed checkpoint. That is intentionally conservative. You can retain several sparse checkpoints, walk backward until all nodes agree, and resume from the common ancestor, but only after defining how deep a rollback your application permits.
Publishing every 10,000th agreed checkpoint through chain_monitor.aleo is a reasonable starting policy. Keep high-frequency state local, anchor selected observations publicly, and store the conversion rule used to turn the checkpoint payload into a Leo field.
Bulk projections change the economics of verification. Pulling three narrow arrays every few seconds is far cheaper than repeatedly downloading full blocks, yet the monitor still checks the pieces that expose stale data and divergent histories. That is enough work for one lion-sized service.