Forward compatible consensus data structures

Abstract

This EIP defines the changes needed to adopt ProgressiveContainer from EIP-7495 and ProgressiveList from EIP-7916 in consensus data structures. Only merkleization changes; serialization of affected types is unchanged.

Motivation

Ethereum’s consensus data structures make heavy use of Simple Serialize (SSZ) Container, which defines how they are serialized and merkleized. The merkleization scheme allows application implementations to verify that neither individual fields nor partial fields have been tampered with. This is useful, for example, in smart contracts of decentralized staking pools that wish to verify that participating validators have not been slashed.

While SSZ Container defines how data structures are merkleized, the merkleization is prone to change across the different forks. When that happens, e.g., because new features are added or old features get removed, existing verifier implementations need to be updated to be able to continue processing proofs.

EIP-7495 ProgressiveContainer is a forward compatible alternative that guarantees a forward compatible merkleization scheme. By transitioning consensus data structures to use ProgressiveContainer, smart contracts that contain verifier logic no longer have to be maintained in lockstep with Ethereum’s fork schedule as long as the underlying features that they verify don’t change. For example, as long as the concept of slashing is represented using the boolean slashed field, existing verifiers will not break when unrelated features get added or removed. This is also true for off-chain verifiers, e.g., in hardware wallets or in operating systems for mobile devices that are on a different software update cadence than Ethereum.

Further, replacing static List and Bitlist capacities with their progressive equivalents decouples operational limits from the SSZ schema. Limits become runtime checks that future EIPs can tune without affecting data types or Merkle proofs. That includes cross-operation constraints such as an overall shared signature-check budget.

Specification

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119 and RFC 8174.

Container conversion

Container types that are expected to evolve over forks SHALL be redefined as ProgressiveContainer(active_fields=[1] * len(type.fields())).

For example, given a type in the old fork:

class Foo(Container):
    a: uint8
    b: uint16

This type can be converted to support stable Merkleization in the new fork:

class Foo(ProgressiveContainer(active_fields=[1, 1])):
    a: uint8
    b: uint16

As part of the conversion, a stable generalized index (gindex) is assigned to each field that remains valid in future forks.

• If a fork appends a field, active_fields MUST be extended with a trailing 1.
• If a fork removes a field, the corresponding active_fields bit MUST be changed to 0.
• Compatibility rules SHOULD be enforced, e.g., by defining a CompatibleUnion[fork_1.Foo, fork_2.Foo, fork_3.Foo, ...] type in the unit test framework.

List[type, N] / Bitlist conversion

List types frequently have excessively large capacities that are never reached in practice, or capacities that have shifted across forks.

• List types with dynamic or unbounded capacity semantics SHALL be redefined as ProgressiveList[type]
• Bitlist types with dynamic or unbounded capacity semantics SHALL be redefined as ProgressiveBitlist

The application logic SHALL be updated to check for an appropriate limit at runtime.

As part of the conversion, a stable generalized index (gindex) is assigned to each list element that remains valid regardless of the number of added elements.

Network message size bounds

For each affected libp2p gossip topic and req/resp chunk type, a constant SHOULD be defined specifying its maximum uncompressed serialized message size, derived from the pre-conversion capacity of the corresponding type. Where no such constant is defined for variable-size messages, MAX_PAYLOAD_SIZE applies.

Converted types

The following types SHALL be converted to ProgressiveContainer:

• Attestation
  • The aggregation_bits field is redefined to use ProgressiveBitlist
• IndexedAttestation
  • The attesting_indices field is redefined to use ProgressiveList
• ExecutionPayloadHeader
• ExecutionPayload
  • The transactions and withdrawals fields are redefined to use ProgressiveList
  • The MAX_TRANSACTIONS_PER_PAYLOAD (1M) limit is no longer enforced
• Transaction is redefined as ProgressiveByteList
  • The MAX_BYTES_PER_TRANSACTION (1 GB) limit is no longer enforced
• ExecutionRequests
  • The deposits, withdrawals and consolidations fields are redefined to use ProgressiveList
• BeaconBlockBody
  • The proposer_slashings, attester_slashings, attestations, deposits, voluntary_exits and bls_to_execution_changes fields are redefined to use ProgressiveList
• BeaconState
  • The validators, balances, previous_epoch_participation, current_epoch_participation, inactivity_scores, pending_deposits, pending_partial_withdrawals and pending_consolidations fields are redefined to use ProgressiveList
• The blob_kzg_commitments, kzg_proofs and column fields are redefined to use ProgressiveList

Immutable types

These types are used as part of the ProgressiveContainer definitions. As they are not ProgressiveContainer themselves, they are considered to have immutable Merkleization. If a future fork requires changing these types in an incompatible way, a new type SHALL be defined and assigned a new field name.

Rationale

Immutability

Once a field in a ProgressiveContainer has been published, its name can no longer be used to represent a different type in the future. This is in line with historical management of certain cases:

• Phase0: BeaconState contained previous_epoch_attestations / current_epoch_attestations
• Altair: BeaconState replaced these fields with previous_epoch_participation / current_epoch_participation

Furthermore, new fields have to be appended at the end of ProgressiveContainer. This is in line with historical management of other cases:

• Capella appended historical_summaries to BeaconState instead of squeezing the new field next to historical_roots
• Electra appended deposit_requests_start_index to BeaconState rather than putting it next to the eth1_deposit_index mechanism that it replaced

With ProgressiveContainer, stable Merkleization requires these rules to become strict.

Validator container

Validator is deliberately kept as Container:

1. It has not changed since genesis; implementations don’t anticipate changes before the post-quantum transition.
2. A post-quantum transition may incorporate a new hash function; such a change would invalidate existing Merkle proofs regardless of whether Validator uses ProgressiveContainer, allowing the transition to restructure the type freely.
3. Converting to ProgressiveContainer adds hashing overhead per validator, which scales to millions of additional hashes.

History accumulators

historical_roots and historical_summaries are deliberately kept as List types. historical_roots has been frozen since Capella, and historical_summaries is actively used by verifiers proving against its hash_tree_root; converting either of them now would add churn or break verifiers without sufficient benefit. A future EIP can address both, potentially consolidating them into a single accumulator alongside related cleanups.

Retroactive application

While ProgressiveContainer / ProgressiveList serialize in the same way as Container / List, the merkleization and hash_tree_root of affected data structures changes. Therefore, verifiers that process historical data predating this EIP still need to support the original merkleization scheme.

Backwards Compatibility

Existing Merkle proof verifiers need to be updated to support the new Merkle tree shape. This includes applicable verifiers in smart contracts on different blockchains and hardware wallets.

Security Considerations

Serialization of affected types and libp2p network message size limits are unchanged by this EIP. Message rejection behavior previously derived from type-specific SSZ bounds is preserved by newly defined per-message bounds constants.

Copyright

Copyright and related rights waived via CC0.