We want to wish the entire community well during the COVID-19 pandemic. While things are turbulent right now, we wanted to share a new post that dives deep into the mempool so that you can dig into an important part of the Web3 ecosystem.

Blockchains are transaction machines. But on-chain data represents value-at-rest — confirmed transactions. While pre-chain, or mempool, data explores value-in motion — transactions in flight. 

To date, the industry has largely focused on value-at-rest. This is an important piece of the Web3 puzzle. But for any transaction to be included in a block, it must move through the mempool. And unlike the blockchain — which acts as a definitive record — the mempool is chaotic, fragmented, and often unpredictable. 

In the course of building infrastructure to monitor the mempool, we've learned a lot about how it works. And at ETHDenver 2020 Matt Cutler, Blocknative's CEO and co-founder, shared some important lessons in his talk Mastering the Mempool:

We've distilled the session into actionable lessons that we believe everyone building in Web3 should know. Let's dive in! 

Experienced builders have been grappling with the complexity of the mempool for some time. Given the structure of the mempool, a number of interesting problems arise: 

• There's no single source of truth.

• Instead, there are thousands of independent mempools, each containing different data.

• Transaction data is constantly in motion.

• The raw data is sparse and sometimes inconsistent.

• This makes definitive mempool data difficult to capture.

• Increasing complexity in transaction architectures compounds these challenges.

While blockchains act as a single source of truth, mempools have no intrinsic form of ensuring consensus. Organizing this chaos into developer-friendly pre-chain data requires substantial resources and expertise. 

When you transact on Web3, you have to hand your transaction over to the network and hope it arrives in a block. Much like checking luggage at the airport. 

The node that receives your transaction broadcasts it out — and you wait for it to get confirmed. If your transaction gets stuck or lost, no central authority can tell you what's going on – or help you resolve the situation.

Given the state of Web3 — and how transactions get validated — transacting creates anxiety. And this anxiety is not felt just by new users. Experienced users, who are more likely to be transacting with real value, often experience the most pronounced transaction anxiety. 

This anxiety is problematic for our ecosystem. And end-users generally do not engage with systems that induce anxiety.

As a community, we spend most of our time talking about value on-chain. And by definition, all on-chain value is at rest. Conversely, the mempool is always in motion. 

This is important because the inner workings of the mempool is not widely understood. And a working knowledge of mempool operations is key to understanding how every transaction comes to rest on-chain. 

Blocknative defines the mempool as:

The shared staging area 'in front' of a blockchain that enables transaction ordering, transaction fee prioritization, and general block construction."

Think of this as the “waiting area” for transactions to be accepted into a block. But depending where you sit in the broader blockchain ecosystem, you might use a different label: 

• Geth calls this the TX-POOL

• Parity calls this the  TX-QUEUE

• Bitcoin calls this the Mempool

• Libra calls this the Mempool

• Tezos calls this the Mempool

Given this, we prefer to use the generic 'Mempool' label for purposes of clarity and inclusiveness. 

There is no canonical, master reference mempool. Instead, there are many mempools. Every node has its own unique mempool. So there are as many mempools as there are nodes. 

A View of The Global Mempool Landscape. Source: matallo.carto.com

The composition of transactions inside of a node's mempool is driven by a number of factors: 

• Configuration settings

• Peering relationships

• Topology

These factors all play a role in determining which transactions enter and persist within a node's mempool. 

The default configuration (slots and memory) settings for Geth and Parity mempools dictate much of their behaviors. 

If all of the slots in a node are full, when a new pending transaction is accepted to the mempool, the transaction with the lowest ETH gas fee will be dropped. Different nodes with different settings will accept and drop different transactions at different rates. 

A missed transaction is a transaction that was written on-chain that your version of the mempool never accepted. From your perspective, this is a transaction that is immediately Confirmed. 

In our analysis, a single node's mempool in a single geography will miss ~1% of all transactions during normal operating conditions. Under heavy transaction volumes, the percentage of missed transactions can vary greatly. 

While on-chain transactions are immutable, in-flight transactions can be overwritten with a replacement transaction. These are commonly referred to as cancel and speed up transactions. 

The percentage of replacement transactions vary over time. While they typically account for ~1% of all transactions, under peak loads replacement transactions can comprise >10% of the mempool. 

During his talk, Matt dives deep into the messy reality of Ethereum transactions. And builders who want to learn more should watch the recording in its entirety or check out the Mastering The Mempool Blog Series.

Notify is a powerful framework that enables developers to access a global node network that captures and organizes the mempool in real-time. Developers can access Notify as an API, SDK, or JS library. This real-time notification network is trusted by over 40 teams in production. 

Developers are leveraging mempool data to reduce development time and debug smart contracts in real-time. By setting up a Notify API data feed in slack or discord, developers can debug Dapps and smart contracts in real-time. This takes about 5 minutes to set up and can empower developers to get instant feedback on their code.

Rather than expecting users to pull data and refresh for updates, your application can leverage the Notify API to push information and let them know what is going on in their transactions. You can see an example of live push notifications in Pillar and Eidoo — both of which alert users of both incoming and outgoing transactions. This feedback loop reduces anxiety for users and helps them to take action when something goes wrong with their transaction.

DeFi protocols can use mempool data to understand how long transactions take on their protocol as compared to other leading protocols. The reality is that many protocols have no insight into transaction times (either the average or when things start breaking). And performance here matters. 

Aside from Matt's talk at ETH Denver, we were excited to see the developer ecosystem dig deep into using Notify to build next-generation applications. 

And coming out of the hackathon, we were excited by one project in particular — OutFront, a service that monitors pending transactions for unauthorized transfers of ERC20 tokens from web3 wallets and fires off a competing transaction at a higher gas price to divert funds into a rescue wallet. The non-custodial system uses a signed permission token (EIP712) to execute the recovery.

Aside from the obvious excitement of seeing a team building with Blocknative present on ETH Denver’s mainstage, we were thrilled to see that the conversation around the complexity of the mempool and the opportunities that pre-chain data can provide.  

The complexity of the mempool has previously prevented teams from building on it.  But now, with Blocknative’s API and SDK, developers can access the rich world of value in motion. And in that world, they can see all of the possible future states of the blockchain.

At any given time, there are tens of thousands to hundreds of thousands of transactions in the mempool. And the number of internal transactions can be two-to-five times the volume of the external transactions. In the mempool, data flows continuously in and out. And about every 15 seconds, the top of the mempool gets included in a block.

This creates a chaotic, ever-changing mempool data environment. As we drive towards 1 billion transactions, it is crucial for our ecosystem to leverage mempool data to improve end-user experience and create more robust experiences. 

As we drive mainstream adoption of blockchain, we need to create friendly end-user experiences. Transaction confidence matters. Transaction performance matters. And the mempool is an important part of the puzzle towards creating a more robust end-user experiences. 

We want to wish the entire community well during the COVID-19 pandemic. While things are turbulent right now, we wanted to share a new post that dives deep into the mempool so that you can dig into an important part of the Web3 ecosystem.

Blockchains are transaction machines. But on-chain data represents value-at-rest — confirmed transactions. While pre-chain, or mempool, data explores value-in motion — transactions in flight. 

To date, the industry has largely focused on value-at-rest. This is an important piece of the Web3 puzzle. But for any transaction to be included in a block, it must move through the mempool. And unlike the blockchain — which acts as a definitive record — the mempool is chaotic, fragmented, and often unpredictable. 

In the course of building infrastructure to monitor the mempool, we've learned a lot about how it works. And at ETHDenver 2020 Matt Cutler, Blocknative's CEO and co-founder, shared some important lessons in his talk Mastering the Mempool:

We've distilled the session into actionable lessons that we believe everyone building in Web3 should know. Let's dive in! 

Experienced builders have been grappling with the complexity of the mempool for some time. Given the structure of the mempool, a number of interesting problems arise: 

• There's no single source of truth.

• Instead, there are thousands of independent mempools, each containing different data.

• Transaction data is constantly in motion.

• The raw data is sparse and sometimes inconsistent.

• This makes definitive mempool data difficult to capture.

• Increasing complexity in transaction architectures compounds these challenges.

While blockchains act as a single source of truth, mempools have no intrinsic form of ensuring consensus. Organizing this chaos into developer-friendly pre-chain data requires substantial resources and expertise. 

When you transact on Web3, you have to hand your transaction over to the network and hope it arrives in a block. Much like checking luggage at the airport. 

The node that receives your transaction broadcasts it out — and you wait for it to get confirmed. If your transaction gets stuck or lost, no central authority can tell you what's going on – or help you resolve the situation.

Given the state of Web3 — and how transactions get validated — transacting creates anxiety. And this anxiety is not felt just by new users. Experienced users, who are more likely to be transacting with real value, often experience the most pronounced transaction anxiety. 

This anxiety is problematic for our ecosystem. And end-users generally do not engage with systems that induce anxiety.

As a community, we spend most of our time talking about value on-chain. And by definition, all on-chain value is at rest. Conversely, the mempool is always in motion. 

This is important because the inner workings of the mempool is not widely understood. And a working knowledge of mempool operations is key to understanding how every transaction comes to rest on-chain. 

Blocknative defines the mempool as:

The shared staging area 'in front' of a blockchain that enables transaction ordering, transaction fee prioritization, and general block construction."

Think of this as the “waiting area” for transactions to be accepted into a block. But depending where you sit in the broader blockchain ecosystem, you might use a different label: 

• Geth calls this the TX-POOL

• Parity calls this the  TX-QUEUE

• Bitcoin calls this the Mempool

• Libra calls this the Mempool

• Tezos calls this the Mempool

Given this, we prefer to use the generic 'Mempool' label for purposes of clarity and inclusiveness. 

There is no canonical, master reference mempool. Instead, there are many mempools. Every node has its own unique mempool. So there are as many mempools as there are nodes. 

A View of The Global Mempool Landscape. Source: matallo.carto.com

The composition of transactions inside of a node's mempool is driven by a number of factors: 

• Configuration settings

• Peering relationships

• Topology

These factors all play a role in determining which transactions enter and persist within a node's mempool. 

The default configuration (slots and memory) settings for Geth and Parity mempools dictate much of their behaviors. 

If all of the slots in a node are full, when a new pending transaction is accepted to the mempool, the transaction with the lowest ETH gas fee will be dropped. Different nodes with different settings will accept and drop different transactions at different rates. 

A missed transaction is a transaction that was written on-chain that your version of the mempool never accepted. From your perspective, this is a transaction that is immediately Confirmed. 

In our analysis, a single node's mempool in a single geography will miss ~1% of all transactions during normal operating conditions. Under heavy transaction volumes, the percentage of missed transactions can vary greatly. 

While on-chain transactions are immutable, in-flight transactions can be overwritten with a replacement transaction. These are commonly referred to as cancel and speed up transactions. 

The percentage of replacement transactions vary over time. While they typically account for ~1% of all transactions, under peak loads replacement transactions can comprise >10% of the mempool. 

During his talk, Matt dives deep into the messy reality of Ethereum transactions. And builders who want to learn more should watch the recording in its entirety or check out the Mastering The Mempool Blog Series.

Notify is a powerful framework that enables developers to access a global node network that captures and organizes the mempool in real-time. Developers can access Notify as an API, SDK, or JS library. This real-time notification network is trusted by over 40 teams in production. 

Developers are leveraging mempool data to reduce development time and debug smart contracts in real-time. By setting up a Notify API data feed in slack or discord, developers can debug Dapps and smart contracts in real-time. This takes about 5 minutes to set up and can empower developers to get instant feedback on their code.

Rather than expecting users to pull data and refresh for updates, your application can leverage the Notify API to push information and let them know what is going on in their transactions. You can see an example of live push notifications in Pillar and Eidoo — both of which alert users of both incoming and outgoing transactions. This feedback loop reduces anxiety for users and helps them to take action when something goes wrong with their transaction.

DeFi protocols can use mempool data to understand how long transactions take on their protocol as compared to other leading protocols. The reality is that many protocols have no insight into transaction times (either the average or when things start breaking). And performance here matters. 

Aside from Matt's talk at ETH Denver, we were excited to see the developer ecosystem dig deep into using Notify to build next-generation applications. 

And coming out of the hackathon, we were excited by one project in particular — OutFront, a service that monitors pending transactions for unauthorized transfers of ERC20 tokens from web3 wallets and fires off a competing transaction at a higher gas price to divert funds into a rescue wallet. The non-custodial system uses a signed permission token (EIP712) to execute the recovery.

Aside from the obvious excitement of seeing a team building with Blocknative present on ETH Denver’s mainstage, we were thrilled to see that the conversation around the complexity of the mempool and the opportunities that pre-chain data can provide.  

The complexity of the mempool has previously prevented teams from building on it.  But now, with Blocknative’s API and SDK, developers can access the rich world of value in motion. And in that world, they can see all of the possible future states of the blockchain.

At any given time, there are tens of thousands to hundreds of thousands of transactions in the mempool. And the number of internal transactions can be two-to-five times the volume of the external transactions. In the mempool, data flows continuously in and out. And about every 15 seconds, the top of the mempool gets included in a block.

This creates a chaotic, ever-changing mempool data environment. As we drive towards 1 billion transactions, it is crucial for our ecosystem to leverage mempool data to improve end-user experience and create more robust experiences. 

As we drive mainstream adoption of blockchain, we need to create friendly end-user experiences. Transaction confidence matters. Transaction performance matters. And the mempool is an important part of the puzzle towards creating a more robust end-user experiences.