Aleo for Agents is offline at present

blog·2026-10-19·10 min read

Project of the Week: Build a Restore-Proven Aleo Node Backup Drill

Introduction

A successful db_backup response proves that snarkOS accepted a backup request. It says nothing about whether the resulting directory can start a node.

That distinction caused a real test bug in snarkOS. PR 4556 found that .ci/test_db_backup.sh wrote checkpoints to checkpoint_<suffix> but restarted validators from ledger_checkpoint_<suffix>. The second directory did not exist, so each validator quietly started from genesis. The supposed rollback assertion still passed.

Ouch.

A useful recovery drill must boot from the directory produced by db_backup. Once running, the restored node needs to report the checkpoint's block height and hash. Application state matters too, because a ledger can look healthy while holding the wrong state for a program your service depends on.

We will build that drill around a tiny Leo canary. The canary stores a monotonically increasing sequence and a caller-supplied marker. We will checkpoint a local snarkOS devnet, inspect the checkpoint through an isolated node, advance the original chain, then start from the same checkpoint again. Recovery passes only when the restored observations match the saved recovery point.

RocksDB checkpoints make the local copy cheap. SST files are immutable, so RocksDB can hard-link them when the checkpoint and ledger share a filesystem. The manifest and mutable log material are copied. Put the destination on another filesystem and those SST files must be copied too.

A local checkpoint is still a local failure domain. Copy or archive it off-host after creation if the drill is meant to cover disk loss.

What we're building

The finished lab has two moving parts:

  • restore_canary.aleo, which stores a sequence and marker in public singleton storage.
  • Shell commands that run a four-validator devnet, create a checkpoint, boot from the written directory, and compare observations with jq.

The comparison file looks like this:

json
{
  "height": 42,
  "block_hash": "ab1...",
  "canary": ["1u64", "1001field"]
}

Height tells us where the restored ledger stopped. The hash proves ledger identity at that height, rather than accepting any chain with the same length. The canary output proves program state was rolled back with the blocks.

I prefer a view fn here instead of scraping RocksDB or inferring state from transaction history. The program owns the meaning of its state, and a view evaluates that state without creating another transaction.

Prerequisites

You need Linux or macOS with these commands on PATH:

  • snarkos built with local development support.
  • Leo 4.4.4 or a compatible 4.4 release.
  • curl, jq, and a POSIX shell with Bash arrays.

Check the installed tools:

bash
snarkos --version
leo --version
curl --version
jq --version

The lab uses the published local development account from Leo's documentation:

bash
export PRIVATE_KEY='APrivateKey1zkp8CZNn3yeCseEtxuVPbDCwSyhGW6yZKUYKfgXmcpoGPWH'

Never point that account at a live network. Everyone knows its private key.

Create the workspace:

bash
mkdir -p restore-drill/restore_canary/src
mkdir -p restore-drill/ops
cd restore-drill
export LAB="$PWD/lab"
export REST='http://127.0.0.1:3030'
mkdir -p "$LAB/live" "$LAB/logs" "$LAB/checkpoints"

Step-by-step

1. Write the Leo canary

Create restore_canary/program.json:

json
{
  "program": "restore_canary.aleo",
  "version": "0.1.0",
  "description": "Public state canary for snarkOS restore drills",
  "license": "MIT",
  "leo": "4.4.4",
  "dependencies": null,
  "dev_dependencies": null
}

Create restore_canary/src/main.leo:

leo
program restore_canary.aleo {
    storage sequence: u64;
    storage marker: field;

    fn advance(public next_marker: field) -> Final {
        return final {
            let current: u64 = sequence.unwrap_or(0u64);
            sequence = current + 1u64;
            marker = next_marker;
        };
    }

    view fn checkpoint_state() -> (u64, field) {
        return (
            sequence.unwrap_or(0u64),
            marker.unwrap_or(0field)
        );
    }

    @noupgrade
    constructor() {}
}

sequence is a storage singleton because there is one global canary position. A mapping with a dummy Boolean key would work, but it would encode a singleton as a collection. That is needless ceremony.

advance returns Final. Its proof-side body carries the public marker into the final {} block, where the node reads and updates public state. Storage access outside finalization would be rejected by Leo.

checkpoint_state is read-only. It returns defaults before the first update, which makes a fresh-ledger mistake visible as ["0u64", "0field"] instead of an HTTP failure.

The empty constructor is mandatory. @noupgrade also freezes the canary's code, which removes upgrades as a source of ambiguity during recovery.

Build it:

bash
cd restore_canary
leo build
leo run advance 1001field
cd ..

leo run checks the function locally, including its inputs and proof-side code. It does not commit the final {} state update to a ledger. We still need a deployed devnet transaction for the recovery test.

2. Add node scripts

Create ops/start-cluster.sh:

bash
#!/usr/bin/env bash
set -euo pipefail

: "${LAB:?export LAB first}"
mkdir -p "$LAB/live" "$LAB/logs"
rm -f "$LAB"/validator-*.pid

for id in 0 1 2 3; do
  rest_port=$((3030 + id))
  snarkos start \
    --validator \
    --nodisplay \
    --dev "$id" \
    --dev-num-validators 4 \
    --ledger-storage "$LAB/live/ledger-$id" \
    --node-data-storage "$LAB/live/node-$id" \
    --rest "127.0.0.1:$rest_port" \
    --nojwt \
    >"$LAB/logs/validator-$id.log" 2>&1 &
  echo $! >"$LAB/validator-$id.pid"
done

until curl -fsS http://127.0.0.1:3030/testnet/block/height/latest >/dev/null; do
  sleep 1
done

Create ops/stop-cluster.sh:

bash
#!/usr/bin/env bash
set -euo pipefail

: "${LAB:?export LAB first}"
shopt -s nullglob

for file in "$LAB"/validator-*.pid; do
  pid=$(cat "$file")
  kill "$pid" 2>/dev/null || true
done

for file in "$LAB"/validator-*.pid; do
  pid=$(cat "$file")
  wait "$pid" 2>/dev/null || true
  rm -f "$file"
done

Create ops/start-restored.sh:

bash
#!/usr/bin/env bash
set -euo pipefail

: "${LAB:?export LAB first}"
: "${CHECKPOINT:?export CHECKPOINT first}"

rm -rf "$LAB/restored-node-data"

snarkos start \
  --client \
  --nodisplay \
  --dev 0 \
  --dev-num-validators 4 \
  --peers "" \
  --ledger-storage "$CHECKPOINT" \
  --node-data-storage "$LAB/restored-node-data" \
  --rest 127.0.0.1:3030 \
  --nojwt \
  >"$LAB/logs/restored.log" 2>&1 &

echo $! >"$LAB/restored.pid"

until curl -fsS http://127.0.0.1:3030/testnet/block/height/latest >/dev/null; do
  sleep 1
done

Create ops/stop-restored.sh:

bash
#!/usr/bin/env bash
set -euo pipefail

: "${LAB:?export LAB first}"

if [[ -f "$LAB/restored.pid" ]]; then
  pid=$(cat "$LAB/restored.pid")
  kill "$pid" 2>/dev/null || true
  wait "$pid" 2>/dev/null || true
  rm -f "$LAB/restored.pid"
fi

Create ops/probe.sh:

bash
#!/usr/bin/env bash
set -euo pipefail

base=${1:-http://127.0.0.1:3030}
out=${2:-/dev/stdout}

height=$(curl -fsS "$base/testnet/block/height/latest")
hash=$(curl -fsS "$base/testnet/block/hash/latest" | jq -r '.')
state=$(curl -fsS -X POST \
  "$base/testnet/program/restore_canary.aleo/view/checkpoint_state" \
  -H 'Content-Type: application/json' \
  -d '[]')

jq -n \
  --argjson height "$height" \
  --arg hash "$hash" \
  --argjson canary "$state" \
  '{height: $height, block_hash: $hash, canary: $canary}' \
  >"$out"

Make the scripts executable:

bash
chmod +x ops/*.sh

The restored node has an empty peer list and runs as a client. It cannot produce blocks, and it cannot sync beyond the recovery point. Isolation matters. A restored node that reconnects immediately may race ahead before you inspect it, hiding the boundary you meant to test.

Current snarkOS calls the ledger option --ledger-storage. The older --storage name is deprecated. Using the explicit name also prevents confusion with node-specific data, which lives under --node-data-storage and may contain validator-sensitive material.

3. Deploy and set the recovery state

Start the validators:

bash
./ops/start-cluster.sh

Deploy from the Leo project:

bash
cd restore_canary
leo deploy \
  --broadcast \
  --devnet \
  --private-key "$PRIVATE_KEY" \
  --endpoint "$REST"

leo execute advance 1001field \
  --broadcast \
  --devnet \
  --private-key "$PRIVATE_KEY" \
  --endpoint "$REST"
cd ..

Poll until the final block has run:

bash
until [[ "$(curl -fsS -X POST \
  "$REST/testnet/program/restore_canary.aleo/view/checkpoint_state" \
  -H 'Content-Type: application/json' \
  -d '[]' | jq -r 'join(",")')" == '1u64,1001field' ]]; do
  sleep 1
done

The read-back is the acceptance condition. A transaction ID proves submission or inclusion, depending on where you observed it. The view proves finalization changed the state we care about.

4. Create the checkpoint

Choose a destination that does not exist:

bash
export CHECKPOINT="$LAB/checkpoints/recovery-$(date -u +%Y%m%dT%H%M%SZ)"
test ! -e "$CHECKPOINT"

Ask node zero to create it:

bash
curl -fsS -X POST \
  "$REST/testnet/db_backup?path=$CHECKPOINT"
printf '\ncheckpoint=%s\n' "$CHECKPOINT"
test -d "$CHECKPOINT"

RocksDB creates the destination directory. Pre-creating it can make checkpoint creation fail, so create only its parent.

The REST call runs with --nojwt because the devnet is bound to loopback. Production nodes should keep JWT authentication enabled and should not expose validator REST ports publicly.

Do not record the live node's current height as the checkpoint height. Blocks can land between the checkpoint operation and your next GET request. We will discover the artifact's boundary by starting from the artifact itself.

5. Establish the recovery point

Stop the live cluster and boot the checkpoint in isolation:

bash
./ops/stop-cluster.sh
./ops/start-restored.sh
./ops/probe.sh "$REST" "$LAB/recovery-point.json"
cat "$LAB/recovery-point.json"
./ops/stop-restored.sh

That file is our recovery contract. It came from the directory written by db_backup, rather than from an estimate taken around the same time.

6. Move the live chain forward

Restart the original validators from their live ledger directories:

bash
./ops/start-cluster.sh

Advance the canary with a different marker:

bash
cd restore_canary
leo execute advance 2002field \
  --broadcast \
  --devnet \
  --private-key "$PRIVATE_KEY" \
  --endpoint "$REST"
cd ..

Wait for the changed state and save it:

bash
until [[ "$(curl -fsS -X POST \
  "$REST/testnet/program/restore_canary.aleo/view/checkpoint_state" \
  -H 'Content-Type: application/json' \
  -d '[]' | jq -r 'join(",")')" == '2u64,2002field' ]]; do
  sleep 1
done

./ops/probe.sh "$REST" "$LAB/advanced.json"
cat "$LAB/advanced.json"

Confirm that the live chain no longer matches the recovery point:

bash
if cmp -s "$LAB/recovery-point.json" "$LAB/advanced.json"; then
  echo 'live chain did not move beyond the checkpoint' >&2
  exit 1
fi

Stop it again:

bash
./ops/stop-cluster.sh

Testing

Start from the checkpoint one more time and capture a fresh observation:

bash
./ops/start-restored.sh
./ops/probe.sh "$REST" "$LAB/restored.json"
cat "$LAB/restored.json"

Compare every field:

bash
jq -e --slurpfile expected "$LAB/recovery-point.json" '
  .height == $expected.height and
  .block_hash == $expected.block_hash and
  .canary == $expected.canary
' "$LAB/restored.json"

For a noisier failure report, run:

bash
diff -u "$LAB/recovery-point.json" "$LAB/restored.json"

A passing result means the node opened the checkpoint directory and returned the same ledger boundary plus the same canary state. The advanced live observation should still contain 2u64 and 2002field, while the restored observation should contain 1u64 and 1001field.

Run explicit assertions if you want the drill to fail with focused messages:

bash
expected_height=$(jq -r '.height' "$LAB/recovery-point.json")
expected_hash=$(jq -r '.block_hash' "$LAB/recovery-point.json")
expected_state=$(jq -c '.canary' "$LAB/recovery-point.json")

actual_height=$(jq -r '.height' "$LAB/restored.json")
actual_hash=$(jq -r '.block_hash' "$LAB/restored.json")
actual_state=$(jq -c '.canary' "$LAB/restored.json")

[[ "$actual_height" == "$expected_height" ]] || {
  echo "height mismatch: expected $expected_height, got $actual_height" >&2
  exit 1
}

[[ "$actual_hash" == "$expected_hash" ]] || {
  echo "hash mismatch: expected $expected_hash, got $actual_hash" >&2
  exit 1
}

[[ "$actual_state" == "$expected_state" ]] || {
  echo "canary mismatch: expected $expected_state, got $actual_state" >&2
  exit 1
}

echo 'restore drill passed'
./ops/stop-restored.sh

Height alone is a weak assertion. A fresh or unrelated chain can eventually reach the same number. Hash alone says nothing about whether your application query works after restart. The canary catches missing program state, failed finalization, and accidental restoration of a different checkpoint.

For another layer of checking, compare the recovered node against an independent node using the range projections from Build a Multi-Node Aleo Chain Consistency Monitor. Keep that comparison bounded at the saved recovery height. A disaster drill should verify the intended rollback point before allowing normal synchronization.

What's next

Move the procedure into scheduled operations once it works locally. Keep checkpoint creation frequent and full restore drills less frequent, since restore tests need spare disk, an isolated port, and enough time to open the database.

Archive recovery-point.json beside the checkpoint. Add the snarkOS version, network, checkpoint path, creation time, and a digest of the archive if you copy it off-host. Those fields explain what an operator is restoring months later.

Keep the checkpoint on the ledger filesystem while creating it if you want cheap hard links. Then copy it to another machine or object store. The off-host copy consumes the full logical size, which is the correct price for surviving loss of the original filesystem.

Finally, test the off-host artifact too. Restoring the convenient local checkpoint proves recovery from process failure. Restoring a transferred archive proves that your copy job preserved filenames, permissions, and database contents. That is the drill I would trust during an incident.

Tags: project-tutorial, snarkos, leo, node-operations, backup, disaster-recovery, testing