A 30-core validator does not need 30 Tokio workers and 30 Rayon workers.

Give both schedulers the whole machine and the process can end up with more runnable threads than CPUs, even before RocksDB, networking, metrics, and operating-system work join the queue. The extra threads do not make consensus finish sooner. They may spend their time stealing tasks, moving between cores, evicting useful cache lines, and waiting on the same shared state.

That is the performance problem behind snarkOS PR #4464 and its companion snarkVM PR #3438. Their architectural point matters more than any one thread count: size each pool from observed demand under a validator workload, not from the number returned by available_parallelism().

I like this change. Core-count defaults are reasonable for isolated runtimes. A validator is not an isolated runtime.

Tokio and Rayon solve different problems.

Tokio runs async tasks. Network reads, timers, channel operations, peer management, and consensus messages can share a small number of workers because well-behaved futures yield when they would otherwise wait. Tokio's multithreaded scheduler uses local queues, a global queue, and work stealing. Its default worker count is the number of CPUs visible to the process.

Rayon handles synchronous CPU parallelism. snarkVM can use it for work that divides into independent pieces and then joins, including expensive verification or ledger operations. Rayon also defaults to the machine's logical CPU count unless the application or RAYON_NUM_THREADS says otherwise.

Each default makes sense on its own. Together, they can lie.

Suppose a validator container sees 30 CPUs. A default Tokio runtime may create 30 async workers while the global Rayon pool creates another 30 workers. Tokio also has an on-demand blocking pool, and the validator has other threads that do not belong to either scheduler. The process can therefore have well over 60 threads competing for 30 execution slots.

Thread count alone is not proof of trouble. Sleeping threads are cheap enough, and an async worker waiting for I/O is not fighting Rayon for cycles. Trouble starts when several pools become runnable during the same block window.

A validator receives a burst of network messages. Consensus wakes tasks to decode and route them. Ledger checks fan out onto Rayon. Storage completes requests and wakes more async work. Metrics scrape at an awkward moment. Linux now has to choose which runnable thread gets a core, and the choice may change before the thread finishes its useful work.

More workers introduce several costs:

• The kernel performs more context switches and CPU migrations.
• Work stealing examines queues even when there is little work to steal.
• Migrated threads lose warm instruction and data caches.
• More tasks can reach shared locks, channels, allocators, or storage queues at once.
• A latency-sensitive consensus task may wait behind throughput-oriented ledger work.

The last item is the nasty one. Average CPU use may look comfortable while block time gets worse. Consensus cares about when a particular chain of work completes, not whether the host averaged 55 percent CPU over a minute.

A CPU core is a shared execution resource. Tokio and Rayon do not negotiate ownership merely because both live in the same Rust process.

Tokio concurrency is also easy to misread. Ten thousand live futures do not require ten thousand workers. If most futures wait on sockets, four workers may keep the queues drained. Adding 26 workers cannot accelerate a packet that has not arrived.

Rayon has a different limit. Thirty Rayon workers help only if the validator can expose roughly thirty useful pieces of CPU work at once. A stage with eight independent jobs cannot keep thirty workers productive. A stage that ends at a barrier may run slower when the scheduler repeatedly moves those eight jobs and their data between cores.

Coordination compounds the problem. Parallel work often ends with a join, a channel send, or an update to shared state. The block pipeline advances when the slowest required job completes. An extra worker that speeds up optional or later work does nothing for that boundary; an extra worker that evicts the slow job's cache can delay it.

A useful model is:

pool size = observed simultaneous demand + a small tested margin

Core count remains a ceiling and a deployment constraint. It is not a demand estimate.

The distinction also explains why validator tuning cannot be reduced to one CPU percentage. Per-thread accounting can show whether six workers carry nearly all Tokio CPU while the rest remain cold. It can reveal a Rayon pool where only a narrow band becomes busy during block processing. Pair those samples with run-queue length, context switches, block time, and achieved TPS. One metric cannot tell the story.

snarkos-stress-testing PR #517 revives an end-to-end setup intended for use before major releases. The stated gates are block-time and TPS regressions. That choice is correct because a validator optimization must improve, or at least preserve, the output users experience.

The first reported runs covered 5-validator and 30-validator devnets on 30-core hosts. Per-thread CPU accounting showed that neither Tokio nor Rayon was the bottleneck in those tested configurations. In other words, the pools were not short of workers. Assigning every visible core to each pool bought no useful capacity.

PR #4464 changes the snarkOS side so Tokio pool sizing follows that measured demand rather than a core-count assumption. PR #3438 applies the same reasoning to snarkVM's Rayon use. The two changes belong together. Shrinking only Tokio while leaving an oversized CPU pool can still let Rayon dominate runnable time; shrinking only Rayon leaves unnecessary async workers participating in scheduling and wakeups.

Exact pool sizes should stay version-specific. I would be suspicious of turning one successful constant into validator folklore. Network topology changes. Consensus code changes. A release may move work across the async and parallel boundary. Hardware also differs in cache layout, simultaneous multithreading, NUMA behavior, and sustained clock speed.

The method is durable:

1. Run a representative validator network, including both smaller and larger validator sets.
2. Drive enough transactions to cover normal load and an intentional saturation period.
3. Record block time, TPS, per-thread CPU, scheduler activity, and queue pressure.
4. Reduce a pool until the first meaningful regression appears, then restore measured headroom.
5. Repeat after changes that move expensive work between snarkOS and snarkVM.

PR #517 matters because it makes that loop repeatable. A benchmark run on one function cannot expose cross-pool contention, network bursts, storage wakeups, or a block-time tail. The reproducible proving benchmark project makes a related distinction: isolate a component when measuring its own improvement, then run an end-to-end workload to learn whether the system became faster.

Operators can inspect the same failure mode without modifying snarkOS. Start with a release-pinned node, a fixed machine shape, and a repeatable transaction workload. Change one pool setting per run.

On Linux, a basic capture can look like this:

PID=$(pidof snarkos)

pidstat -t -u -p "$PID" 1
pidstat -w -p "$PID" 1
perf stat -p "$PID" \
  -e context-switches,cpu-migrations,cycles,instructions,cache-misses \
  -- sleep 120

Thread names and exact observability depend on the build, so retain the raw thread IDs as well. Correlate the capture with block timestamps and submitted transaction counts. A pool is oversized when reducing it leaves block time and TPS unchanged while lowering scheduler activity or freeing CPU for the rest of the node.

Consider a hypothetical result on the same 30-core host:

Those numbers are illustrative, not results from the PRs. The middle configuration wins because it preserves throughput and improves the block-time tail. The smallest configuration crossed a real capacity boundary. Picking the row with the fewest threads would be as lazy as picking the row with the most.

Watch for false wins. A five-minute test may miss epoch-related work, peer churn, compaction, or a burst of certificate verification. CPU quotas can also make the process see a different topology from the capacity it receives. Run long enough to hit storage maintenance and network disruption, then repeat the same workload several times.

Telemetry must stay out of the timing path while you measure. The earlier article on removing telemetry from the BFT hot path covers the lock hazard: observation code can alter the schedule it is meant to observe. Per-thread operating-system counters are useful here because they do not require consensus tasks to share a new application lock.

Validator operators should treat the shipped settings as tested defaults, not invitations to map every thread pool to every core. Avoid setting TOKIO_WORKER_THREADS or RAYON_NUM_THREADS from a generic tuning guide unless a pinned snarkOS release documents that override and your own load test supports it.

Container deployments need extra care. CPU affinity, cpusets, quotas, and NUMA placement affect what the runtime sees and what the kernel can schedule. A process that detects 30 logical CPUs but receives a quota equal to eight CPUs can build pools for capacity it does not own. Pinning can help repeatability, though careless pinning can put consensus workers and storage threads on the same small set of cores.

Developers should keep CPU-heavy work away from Tokio workers. spawn_blocking prevents synchronous work from freezing an async worker, but its pool can grow far beyond the useful CPU parallelism of a validator. Sustained computation belongs in a bounded pool whose size was tested with the rest of the process. Long-lived loops deserve dedicated ownership rather than permanent occupancy in a general pool.

Agents that tune validators should optimize against constraints, not thread counts. Give the agent block-time percentiles, TPS, missed rounds, context switches, CPU migrations, memory, and per-thread residency. Require repeated runs and reject any configuration that improves average throughput while damaging the block-time tail.

The larger architectural lesson is pleasantly plain. A validator is a coordinated pipeline. Faster local scheduling is useful only when it shortens the path that produces the next block. PR #4464, PR #3438, and the revived stress harness replace a hardware proxy with measurements of that path. That is how runtime configuration should be chosen.