A peer claims that a collection contains three entries. The decoder reads A, B, then B again and inserts them into a set. The resulting collection has two entries.

Nothing crashed. Every entry parsed. The set behaved exactly as designed.

The decoder did not.

ProvableHQ's snarkVM PR #3410 fixes that boundary in AleoBFT's BatchHeader, BatchCertificate, and Subdag codecs. Each format carries one or more count-prefixed collections. Before the patch, repeated entries could disappear during insertion into an IndexSet or overwrite an existing entry in a BTreeMap. The decoder returned an object whose collection contained fewer distinct items than the encoded count had promised.

That sounds like bookkeeping. In consensus code, it is a canonical-encoding bug with teeth.

Consider a simplified set encoding:

count:u32 || entry_0 || entry_1 || ... || entry_(count-1)

A count of 3 means three encoded entries follow. The decoder must account for all three as distinct members if the destination type is a set. Otherwise, several byte strings can decode to the same object:

02 | A | B
03 | A | B | B
04 | A | A | B | B

A permissive decoder maps every line above to {A, B}. The serializer can emit only the first form because the in-memory set has two members:

Encode(Decode(03 | A | B | B)) = 02 | A | B

Accepted bytes no longer survive a decode-encode round trip. More formally, the codec has lost the property Aleo wants at consensus boundaries:

For every accepted byte string b: Encode(Decode(b)) = b

Some formats intentionally normalize their input. JSON parsers may ignore whitespace, for example. AleoBFT's binary messages have no need for duplicate spellings. Raw message bytes may meet hashes, caches, signatures, network deduplication, logs, or storage indexes before every component agrees on the decoded object.

The earlier post on unconsumed nested bytes covered another version of the same codec failure. A frame declared five bytes while its child parser consumed four. Here, a count declares three entries while the destination collection retains two. Both bugs let the wire claim more structure than the decoded value contains.

AleoBFT builds a Narwhal-style DAG and applies Bullshark-style ordering rules to it. Certificates are DAG vertices. References to certificates in the previous round form edges. Validators gather acknowledgements from distinct committee members before a batch becomes certified.

Distinctness is protocol data.

A BatchHeader carries the batch author's proposal, including transmission identifiers and references to earlier certificates. Those identifiers naturally belong in set-like collections because repeating one does not create a second transmission or a second DAG edge.

A BatchCertificate combines a batch header with the signatures or signer-related entries that certify availability. Quorum reasoning concerns distinct validators. Copying the same encoded member twice cannot manufacture another validator.

A Subdag groups certificates used for a committed portion of the DAG. Its representation uses ordered map and set structures so rounds and certificates have stable iteration behavior. Repeating a certificate does not create another vertex. Repeating a map key is worse: ordinary BTreeMap::insert replaces the value already stored under that key.

PR #3410 addresses the deserialization paths for these structures. The unsafe shape looked like this:

let mut transmission_ids = IndexSet::new();
for _ in 0..num_transmission_ids {
    transmission_ids.insert(TransmissionID::read_le(&mut reader)?);
}

IndexSet::insert returns false when the value already exists. Discarding that Boolean discards the only immediate proof that the wire representation lied about cardinality.

Map insertion has a similar signal:

let previous = rounds.insert(round, certificates);

For a BTreeMap, previous.is_some() means the encoded key occurred earlier. Continuing would silently choose the later value while erasing the earlier one.

The safe pattern checks insertion at the moment the decoder still knows which encoded entry caused the collision:

let id = TransmissionID::read_le(&mut reader)?;
if !transmission_ids.insert(id) {
    return Err(error("duplicate transmission ID"));
}

For a map:

if rounds.insert(round, certificates).is_some() {
    return Err(error("duplicate round"));
}

Those snippets show the pattern rather than the patch's exact error plumbing. The architectural rule is the useful part: insertion failure during decoding is malformed input, not a harmless collection detail.

A decoder could read every entry, build the collection, then compare collection.len() with the prefix. That catches exact duplicate set members in a compact way:

if transmission_ids.len() != num_transmission_ids {
    return Err(error("collection cardinality mismatch"));
}

I prefer checking each insertion.

Immediate rejection keeps the invariant local: after iteration i, the collection has gained exactly i + 1 members. The error also identifies the operation that failed rather than reporting a mismatch after the source entry has gone out of scope.

Maps make the case stronger. A repeated key can replace its previous value without reducing final map length relative to some surrounding expectation unless the decoder compares against the original pair count. Checking insert catches both an identical key-value pair and a repeated key carrying a different value. The latter is not mere compression. It gives the later entry authority to rewrite data already parsed from the same message.

Collecting through an iterator is risky for the same reason:

let values: IndexSet<_> = (0..count)
    .map(|_| Value::read_le(&mut reader))
    .collect::<Result<_, _>>()?;

Parsing errors propagate. Duplicate errors do not exist because FromIterator for a set defines duplicates as normal input. That behavior is good for application code and wrong for a strict wire grammar.

Generic collection helpers should therefore expose a uniqueness policy. A helper named read_counted_set ought to reject a second equal member by default. Callers that want deduplication should ask for it with a different API whose name makes normalization obvious.

Narwhal advances after gathering certificates from a quorum of distinct validators. Aleo's description expresses the threshold as n - f, where f is the largest tolerated Byzantine population below one third of the committee. In the familiar n = 3f + 1 case, that is 2f + 1.

A count prefix cannot stand in for that distinctness check. A message containing 67 serialized signer entries might retain only 45 members after duplicates collapse. Any later quorum check that uses the retained set sees 45, while telemetry or an earlier framing check may still report 67 encoded entries.

Good validation needs both facts:

• The wire contained exactly the number of distinct entries it declared.
• The retained entries satisfy committee membership, signature validity, stake thresholds, round rules, and other semantic conditions.

Rejecting duplicates does not replace certificate verification. It prevents the parser from changing the evidence before certificate verification begins.

The same reasoning applies to DAG edges. AleoBFT requires references to certificates from distinct validators in the previous round. Repeating one certificate ID cannot add another edge in the graph-theoretic sense. If the decoder erases repeats, downstream code receives a tidier graph than the peer sent. Consensus validation should examine the peer's actual claim, not a cleaned-up version of it.

Canonical encoding is often explained as protection against hash malleability. Two byte strings decode to one object, yet hashing the raw bytes gives two digests. That risk is real, though it is only part of the problem.

Different implementations may disagree about duplicate handling. One rejects the message. Another keeps the first map value. A third keeps the last. A streaming implementation may pass both entries to application code. RFC 8949's discussion of duplicate CBOR map keys documents these exact failure modes and recommends rejection when a protocol's data model requires unique keys.

Cross-implementation disagreement is especially unpleasant for agents that inspect, relay, or reconstruct AleoBFT messages. An indexing agent might normalize a malformed certificate and store the canonical result. A relay might preserve the original bytes. A debugger then sees the same semantic identifier attached to different payload digests, while both systems insist they handled one object.

Resource accounting can drift too. A prefix of 100 causes 100 decode operations even if every entry is equal and the final set has one member. Existing count limits still cap the work, which is why hard limits on untrusted edges remain necessary. Duplicate rejection adds a separate guarantee: retained cardinality matches encoded cardinality.

Canonicality has already appeared elsewhere in snarkVM. The point-at-infinity fix rejected a nonzero coordinate payload that the infinity branch previously ignored. PR #3410 applies the same discipline to collection structure. A decoder must not erase meaningful input and then claim it parsed one canonical object.

Round-trip tests starting from valid objects will miss this bug. The serializer cannot create duplicates from a set, so generated objects exercise only canonical encodings.

Adversarial tests must start with bytes. For every count-prefixed set or map, I would require these cases:

• A canonical collection decodes and re-encodes byte for byte.
• Repeating the first, middle, or final set member fails.
• A repeated map key with the same value fails.
• A repeated map key with a different value fails.
• A zero-length collection follows the field's stated minimum rules.
• A prefix above the configured maximum fails before large allocation or long iteration.
• Property tests assert that every successfully decoded input satisfies encoded_count == retained_len.
• Differential tests feed the same malformed corpus to any Rust, TypeScript, WASM, or agent-side parser expected to understand the format.

Fuzzers need a duplicate-aware mutator. Pick one encoded entry, copy it into another slot, increment the count when needed, and preserve enough framing for the parser to reach the insertion. Random byte mutation rarely produces a fully valid cryptographic object twice. Structural mutation does so on demand.

Code review deserves one blunt rule as well: every deserializer that inserts untrusted entries into a uniqueness-enforcing collection must inspect the insertion result. Search for ignored calls to insert, iterator collect operations targeting sets, and map construction where duplicate keys overwrite values.

PR #3410 puts the rejection at the right layer. BatchHeader, BatchCertificate, and Subdag validation now receive the structure the peer encoded or no object at all. A prefix claiming k entries produces exactly k retained entries.

That is what the count always meant.