Substrate blockchain

Substrate/Polkadot-SDK:

The modularity and expandability allows Substrate to be highly customizable and innovative, making it one of the most advanced technologies in the blockchain world. Substrate's modularity comes from its architecture, which allows to customize, adapt or develop components that are specifically needed for a blockchain. This flexibility means that developers don't have to start from scratch when building a blockchain; instead, they can draw on predefined modules that include features such as consensus mechanisms, governance models and user interfaces.

The distinction between client (sc_), primitive (sp_) and FRAME (_frame _pallet) libraries in substrates is an important aspect of understanding the architecture and functionality of substrate-based blockchains. In the context of consensus mechanisms such as Aura and GRANDPA, this distinction can be illustrated clearly.

Example:

• sc_aura: Implements the network side of the Aura consensus mechanism, including the node logic for block production according to the Aura protocol.
• sp_aura: Provides the interfaces and type definitions needed to implement the Aura consensus within the runtime. This is the "logic" side of Aura, which defines how blocks are generated using the Aura protocol.
• sc_grandpa: Responsible for the integration of the GRANDPA consensus into the Substrate Node, in particular for the network logic required to support the final block confirmation by GRANDPA.
• sp_grandpa: Contains the definitions and helper functions that are necessary for the integration of GRANDPA into the blockchain runtime, especially for the logic of the final block confirmation.

It can be imagined in this way, the primitive modules are associated with the "theory" or definition of how the blockchain should work, while the client modules deal with the "practice", i.e. how the blockchain works in real network operation.

There are also pallet_aura and pallet_grandpa which also provide additional consensus functionalities. In summary, it can be said that the sp_ libraries provide basic types and interfaces (primitives), the sc_ libraries are responsible for client implementation, and the pallet_ libraries implement the specific consensus logic within the blockchain runtime.

At first this can seem confusing. However, each of these libraries serves a specific purpose within the ecosystem and together they contribute to the implementation and smooth functioning of the respective consensus mechanism. This separation allows to flexibly configure and adapt the consensus mechanism.

Substrate blockchain example

Demo blockchain network data

• Blockchain technology: [Substrate/Polkadot-SDK]
• Blockchain layer: L1 (standalone)
• Blockchain network: [Demo-network]
• Blockchain network nodes: [Alice-Validator] [Bob-Validator]
• Blockchain network consensus: [Aura] [GRANDPA]
• Blockchain network sudo authority: [Alice]
• Blockchain network infrastructure logic: [substrate client]
• Blockchain network application logic: [substrate runtime]

Demo blockchain substrate modules

Blockchain infrastructure logic modules for the client:

• substrate client
  • sc-basic-authorship: It is used to create and prepare new blocks, which are then proposed to the network for validation and inclusion in the blockchain.
  • sc-cli: Enables node operators and developers to perform various operations and configurations via the command line, e.g. starting a node or adjusting network settings.
  • sc-client-api: Provides developers with access to core functions of the Substrate client, such as querying the blockchain state or sending transactions.
  • sc-consensus: Enables the integration and management of different consensus algorithms in a Substrate blockchain.
  • sc-consensus-aura: Aura is used specifically for the regular and predictable production of blocks in a blockchain network.
  • sc-consensus-grandpa: Enables blockchains to use GRANDPA for the efficient and secure finalization of blocks.
  • sc-executor: Executes the code of smart contracts and other runtime functions that run on the blockchain.
  • sc-network: Enables communication and data exchange between the nodes of a blockchain, including the transfer of blocks and transactions.
  • sc-offchain: Used for operations that need to be performed outside of the blockchain, such as querying external data sources or performing lengthy calculations.
  • sc-rpc-api: Allows external applications and services to interact with a Substrate blockchain by sending requests and receiving responses.
  • sc-service: Used to configure and start a Substrate node, including the integration of consensus mechanisms and network functions.
  • sc-transaction-pool: Organizes and prioritizes transactions before processing and recording them in blocks.
  • sc-transaction-pool-api: Provides an interface to inspect the transaction pool, add or remove transactions and query the status of waiting transactions.
• substrate primitives
  • sp-api: Used by developers to create applications that interact with the blockchain by querying data or sending transactions.
  • sp-blockchain: Used to process and validate blocks and to store and retrieve information on the blockchain.
  • sp-block-builder: Allows developers to construct blocks with transactions and other necessary information before adding them to the blockchain.
  • sp-core: Contains essential tools and components used in many areas of Substrate development, such as cryptography and data management.
  • sp-consensus-aura: Enables the integration of the Aura consensus mechanism into the blockchain runtime to support regular block production by a set of validators.
  • sp-consensus-grandpa: Used for the implementation of Grandpa for fast and secure finalization of blocks in the blockchain runtime.
  • sp-inherents: Used to process information such as timestamps or other block information that does not originate from transactions.
  • sp-io: Enables access to system functions, such as random numbers or time measurement, which are required for the execution of the blockchain runtime.
  • sp-keyring: Simplifies the creation and management of key pairs for developers and users interacting with Substrate blockchains.
  • sp-runtime: Includes essential components such as type definitions, cryptography functions and support for randomization that are required in the blockchain runtime.
  • sp-timestamp: Enables the blockchain to process and store time information, which is important for many applications and processes within the blockchain.
  • substrate-frame-rpc-system: Enables external applications and tools to query important system information via the blockchain, such as block height, system version or active nodes.
• substrate FRAME pallets
  • frame-system: It manages core aspects of the blockchain such as block times, block numbers and transaction counts. It serves as the backbone of any Substrate-based blockchain by providing essential frameworks and functionalities on which other modules can build.
  • pallet-transaction-payment: makes it possible to configure and collect transaction fees. It ensures that transactions performed on the network can be subject to fees.
  • pallet-transaction-payment-rpc: This module extends the functionalities of Substrate and FRAME by allowing external developers and applications to interact directly with the pallet-transaction-payment via Remote Procedure Calls (RPC).
• build process
  • substrate-build-script-utils: is an important tool in a Substrate developer's toolkit that helps to make the build process more efficient and avoid common pitfalls.

Blockchain application logic modules for the runtime

• substrate primitives
  • sp-api: Used by developers to create applications that interact with the blockchain by querying data or sending transactions.
  • sp-block-builder: Allows developers to construct blocks with transactions and other necessary information before adding them to the blockchain.
  • sp-core: Contains essential tools and components used in many areas of Substrate development, such as cryptography and data management.
  • sp-consensus-aura: Enables the integration of the Aura consensus mechanism into the blockchain runtime to support regular block production by a set of validators.
  • sp-consensus-grandpa: Used for the implementation of Grandpa for fast and secure finalization of blocks in the blockchain runtime.
  • sp-genesis-builder: Enables the configuration of the initial state of the blockchain at initialization, including the distribution of tokens and the setup of parameters.
  • sp-inherents: Used to process information such as timestamps or other block information that does not originate from transactions.
  • sp-offchain: Used for operations that take place without direct influence on the blockchain, e.g. collecting external data or interacting with web services.
  • sp-runtime: Includes essential components such as type definitions, cryptography functions and support for randomization that are required in the blockchain runtime.
  • sp-session: Is used to determine which validators are responsible for the upcoming blocks and when a change takes place.
  • sp-std: Provides a compatible interface to Rust's standard library and adds specific extensions for blockchain development.
  • sp-storage: Provides functions and types for storing and querying data in the blockchain. Used to efficiently manage and query the state of the blockchain.
  • sp-transaction-pool: Organizes pending transactions, prioritizes them and prepares them for inclusion in upcoming blocks.
  • sp-version: Used to communicate the current version of the runtime, including version numbers, build information and other relevant metadata.
• substrate FRAME pallets
  • frame-executive: It coordinates the various steps necessary to process transactions, including the verification, execution and finalization of transactions. It ensures that all necessary checks and balances are executed.
  • frame-support: Used by developers to create new blockchain functionalities in the form of modules/pallets by providing frequently needed functions such as storage access and event notifications.
  • frame-system: It manages core aspects of the blockchain such as block times, block numbers and transaction counts. It serves as the backbone of any Substrate-based blockchain by providing essential frameworks and functionalities on which other modules can build.
  • frame-system-rpc-runtime-api: Enables external applications and services to query important information about the blockchain, such as the current block number or system configuration.
  • pallet-aura: Used in blockchains that require a predictable and consistent rhythm of block creation by rotating a fixed group of validators.
  • pallet-balances: Enables the management of token balances, transfers between accounts and other financial-related operations within the blockchain.
  • pallet-grandpa: Enables efficient and secure finalization of blocks, which means that once finalized, blocks are considered final and unchangeable.
  • pallet-sudo: Mainly used in development and test networks to make quick changes or upgrades without going through the usual governance process.
  • pallet-timestamp: Important for many applications and processes that require precise time measurement, such as determining the sequence of events or the execution of actions at specific times.
  • pallet-transaction-payment: Allows to calculate, adjust and distribute fees for the execution of transactions, which is an important source of revenue for validators and a measure to prevent spam on the network.
  • pallet-transaction-payment-rpc-runtime-api: Allows external users and services to get information about transaction fees, such as the estimation.
• substrate WASM builder
  • substrate-wasm-builder: is a tool used within the Substrate development framework to automate and simplify the WebAssembly (WASM) compilation of the runtimes.

How it works?

To show how a process looks like on the demo blockchain, we will use a scenario where Alice (account) will transfer a certain number of units to Bob (account).

Transaction generation and submission

• Transaction data: Alice creates a transaction with the following data: Recipient address (Bob), quantity (200 units), nonce (a number that makes each of her transactions unique), and a validity period, if applicable.

• Fee calculation: The system calculates transaction fees based on factors such as "blockweight" (a measure of the transaction's resource usage) and network utilization. These fees are used to compensate the validator for processing the transaction.

• Signing: Alice signs the transaction with her private key to prove its authenticity.

• Submission to the transaction pool: The signed transaction is sent to the transaction pool, where it waits to be included in a block. The pool prioritizes transactions based on fees and other factors.

More details:

Transaction fee calculation: The fees for a transaction in Substrate are determined by a model that takes several factors into account, including the "blockweight". The blockweight is a measure of the complexity and resource utilization of a transaction. It helps the system to regulate the capacity of a block by ensuring that the execution of transactions within a block does not exceed a predefined time limit.

Formula for calculating fees: The actual cost of a transaction is often calculated as a combination of a base fee, a variable fee (depending on the block weight) and a transaction length fee (based on the size of the transaction in bytes). Substrate also allows these calculations to be customized by the runtime module.

Blockweight limits: Each block has a maximum weight limit that prevents the system from being overloaded by too many or too complex transactions. If a transaction would exceed this limit, it cannot be included in the current block and must wait to be included in a future block.

Blocking production process (Aura)

• Validator selection: At each slot (a time window for block creation), the Aura consensus algorithm deterministically selects a validator from the set of authorized validators.

• Block creation: The selected validator checks the transaction pool, selects transactions (taking into account transaction fees, nonce and other criteria) and creates a new block. In addition to the transactions, the block also contains the state root hash, which represents the state of the blockchain after all transactions in the block have been executed.

More details:

The validator selected by Aura for the current slot looks at the transaction pool. Each transaction in the pool has a weight, and the validator must make a selection to compile the optimal set of transactions without exceeding the maximum block weight. This requires efficient computation as the selection must be made in a way that maximizes block usage and optimizes transaction fees.

Block distribution

The newly created block is immediately distributed to the entire network so that other nodes can check it and add it to their blockchain.

Block validation and finalization (GRANDPA)

• Preliminary validation: Other validators and full nodes check the block for correctness by validating transaction signatures and ensuring that the state root hash is correct.

• Voting by GRANDPA: GRANDPA allows validators to vote for the finalization of blocks. This voting is independent of block creation, which enables fast finalization even during temporary network disruptions.

• Block finalization: Once enough votes have been collected for a block and its predecessors, the block (and all predecessor blocks) is considered finalized. A finalized block can no longer be changed or removed.

More details:

GRANDPA Finalization: After a block has been created and distributed, GRANDPA comes into play to facilitate finalization. GRANDPA is separate from the process responsible for block production (in this case Aura) and instead focuses on achieving network consensus on the finality of blocks. This is done through validator reconciliation.

Weight and finalization: Although the block weight primarily influences block creation, it also plays a role in finalization, as it ensures that only validated and efficiently packed blocks are finalized. GRANDPA considers blocks finalized when a supermajority (usually 2/3) of the votes are reached. GRANDPA's efficiency and speed in the voting phase allows it to achieve finality for blocks quickly, even if some validators are temporarily unavailable.

Immutability through finalization: Once a block has been finalized by the GRANDPA consensus mechanism, it is considered final and immutable. This means that the transactions contained in this block, including the transfer from Alice to Bob, cannot be reversed.

Updating the account balances

• Execution of the transaction: When the block is finalized, the transaction from Alice to Bob is officially executed. Substrate's Runtime executes the transaction, reducing Alice's account balance by 200 units plus the transaction fees and increasing Bob's account balance by 200 units.

• Update of the state: The final state is stored in the blockchain, including Alice's and Bob's updated account balances.

More details:

State update: The blockchain updates its state to reflect the new account balances. The exact calculation of the new state is done by executing the runtime code that is responsible for processing the transaction. Alice's balance is reduced by 200 units plus the transaction fees, while Bob's balance is increased by 200 units.

Storage and state: After the transaction has been executed and the account balances have been updated, the new state is permanently stored in the blockchain. This process involves saving the new state of Alice and Bob's account balances and the updated state root hash, which is a feature of all states in the blockchain.

Balance update: The exact logic of how balances are updated is implemented in the Substrate's Balances module. This module takes care of the core functionality of balances within the blockchain, including credits (deposits) and debits (withdrawals).

Confirmation and notification

• Confirmations: Once the block is finalized, Alice and Bob (and anyone else watching the blockchain) can consider the transaction confirmed.

• Notification: Users can be notified of changes in the blockchain through their wallets or blockchain explorers that reflect the new state.

The transfer of 200 units from Alice to Bob in a Substrate-based blockchain involves many steps that range from the initial creation and signing of the transaction, through block creation and finalization, to execution and final storage in the blockchain state. Each step is enabled by the interaction of different layered mechanisms and modules within the Substrate framework, including the Aura and GRANDPA consensus mechanisms, the Balances module for balance management, the use of blockweight and transaction fees to regulate network utilization and security etc.