Subscribe to Gas Network - Blog
>100 subscribers
Subscribe to Gas Network - Blog
>100 subscribers
Share Dialog
Blocknative has uncovered evidence that the MakerDAO liquidations on March 12 and 13 were an engineered event. This evidence is derived from the mempool, the pre-chain area miners use to create blocks. The mempool is an often overlooked – and commonly under-appreciated – aspect of the Ethereum ecosystem.
As mempool specialists, Blocknative operates a global network of Geth and Parity (OpenEthereum) nodes in a variety of configurations. This infrastructure powers our real-time mempool monitoring services so that we can capture, normalize, and archive the otherwise ephemeral state changes of the mempool. Blocknative captured over 30 million rows of data over the course of Black Thursday, enabling an open-ended investigation that has so far uncovered several 'vulnerabilities' that appear to have been exploited.
While much has been written about the outcomes of Black Thursday, in this post we will publicly explore data from Blocknative’s Mempool Archive for the first time, bringing both new data and theories to the process. We are also making our underlying forensic mempool data set for this time period available to the ecosystem for others to review (see below).
If you are a security researcher who would like to learn more about our findings, or to explore other potential mempool anomalies, please get in touch.
Blocknative's forensic mempool investigation surfaced three major contributing mempool factors to the events on March 12 and 13:
Stuck Transactions
Share Dialog
Blocknative has uncovered evidence that the MakerDAO liquidations on March 12 and 13 were an engineered event. This evidence is derived from the mempool, the pre-chain area miners use to create blocks. The mempool is an often overlooked – and commonly under-appreciated – aspect of the Ethereum ecosystem.
As mempool specialists, Blocknative operates a global network of Geth and Parity (OpenEthereum) nodes in a variety of configurations. This infrastructure powers our real-time mempool monitoring services so that we can capture, normalize, and archive the otherwise ephemeral state changes of the mempool. Blocknative captured over 30 million rows of data over the course of Black Thursday, enabling an open-ended investigation that has so far uncovered several 'vulnerabilities' that appear to have been exploited.
While much has been written about the outcomes of Black Thursday, in this post we will publicly explore data from Blocknative’s Mempool Archive for the first time, bringing both new data and theories to the process. We are also making our underlying forensic mempool data set for this time period available to the ecosystem for others to review (see below).
If you are a security researcher who would like to learn more about our findings, or to explore other potential mempool anomalies, please get in touch.
Blocknative's forensic mempool investigation surfaced three major contributing mempool factors to the events on March 12 and 13:
Stuck Transactions
Mempool 'Compression' — a dramatic reduction in the marketable portion of the mempool (that is, transactions with sufficient gas to be considered by miners), which can bias ETH gas price estimates.
'Hammerbots' — Automated transaction systems that magnified congestion, and hence compounded mempool compression.
We are publishing this analysis and the associated data set in efforts to raise awareness of the dynamic transaction risks that can rapidly emerge during periods of Blockchain network congestion.
On March 12, 2020, the crypto markets saw one of their biggest sell-offs in history, with the price of ETH falling 43% – and BTC falling 39% – in a matter of hours. As prices fell, a negative feedback loop emerged that reduced liquidity and forced liquidations from various DeFi contracts. This downward pressure created real challenges for every entity attempting to transact on the network during this period.
Hourly weighted average ETH prices from March 12 through March 13, 2020 UTC. Source: cointelegraph.com.
The rapid price change caused sustained congestion in the Ethereum mempool, as automated transaction systems, such as trading bots, programmatically reacted to the volatility.
One negative consequence of this congestion were 'zero bid auctions' on liquidated MakerDAO CDPs. Of the 3,994 liquidation auctions associated with Black Thursday, 1,462 or 36.6% were won by zero bids. Over a roughly 12 hour period, $8.32 million in aggregated locked CDP value was lost to these zero bid auctions. More on this below and via MakerDAO's original The Market Collapse of March 12-13, 2020: How It Impacted MakerDAO post.
The rapid price change in ETH on March 12 created congestion in the mempool. This was exacerbated by price-triggered bot activity.
Hourly count of new pending transactions arriving into the mempool versus confirmed transaction count.
The result was a surge of transactions propagating through the network which forced nodes with default mempool configurations to actively protect their system resources by:
Evicting, or dropping, many valid transactions.
Rejecting, or ignoring, many valid transactions.
Mempools generally evict the transactions with the lowest associated fees. And although these transactions are not truly ‘lost’ across the entire network, evicted transactions may be significantly delayed until network conditions normalize.
Hourly count of evicted, or dropped, transactions along with new stuck (not processable) transactions arriving for March 11, 2020 through March 14, 2020.
Critically, evictions have a side effect of increasing the likelihood of stuck transactions.
When a transaction is evicted, the node does not remember any details about the transaction, such as the sending address or the transaction nonce. Therefore, when a new transaction arrives with the next nonce, the node may detect a nonce gap and will be unable to process the new transaction. Impacted transactions are then placed in the unprocessable Queued portion of the node's mempool. These transactions become stuck regardless of their associated transaction fees (that is, gas prices).
For more on this, please see our prior post covering transaction ordering: A Note On Nodes: Your Gateway To The Mempool.
Meanwhile, network congestion caused rapid increases in the minimum gas price needed to be admitted to a mempool, so the original transactions were now rejected as underpriced. And since the previously evicted transactions were unable to get back into the mempool, they could not span the transaction nonce gap. In fact, some node implementations actively ignore these rejected transactions for a while in efforts to prevent spamming over the peer-to-peer network. The nonce gap effectively locked addresses out of completing new transactions.
Stuck transactions were more likely to impact addresses that issue many new transactions, including automated trading systems, payment networks, and possibly even exchanges. These systems attempted to course correct, often compounding congestion as more and more transactions are delayed.
The only solution? You must first proactively detect when one of your transactions might be stuck. This can be challenging, as your transaction may only be stuck on some – but not all – nodes on the network. Then, you must identify the first missing transaction in the nonce gap and immediately speed up that transaction with a marketable gas price. Finally, you must continue to monitor the original stuck transaction to ensure that it has been moved from the Queued to the Pending portion of the mempool and has successfully proceeded to finalization. If your transaction remains stuck, repeat the process above until you can confirm that all transactions in the nonce gap have been successfully resolved. And yes, this is one of the many reasons why we built and operate our Notify API.
The increase in network congestion – and the resulting increase in stuck transactions – caused the marketable portion of the mempool to shrink. We call this 'mempool compression'.
Miners are incentivized to maximize revenue through block reward and gas utilization at the highest price. Mining operations make decisions on which block templates – that is, candidate blocks – to mine based on the offered gas prices of transactions in the mempool.
Due to congestion, several behaviors contributed to compressing the mempool so that a much smaller portion remained relevant to miners:
Exhausted resources: Depending on node implementation, the dramatic increase in stuck transactions consumed substantial mempool resources. This in turn reduced available node resources for processing valid pending transactions.
Escalating gas price: The falling ETH price triggered many 'high urgency' transactions that were structured to be confirmed before the value of ETH fell even lower. This drove both individual users and automated bots to rapidly escalate gas prices. Since not all participants were paying close attention to the congestion, the aggregate profile of pending mempool transactions included an unusually large number of underpriced – and hence unmarketable – entries.
Insufficiently responsive pricing algorithms: The gas price distribution of pending transactions in the mempool, and of those confirmed in recently mined blocks, could distort gas price Oracle calculations. This effectively underpriced gas for broad swaths of new transactions, further exacerbating pending transaction gas price disparities.
As mempool congestion rose, the gas price required to be mined and get on-chain climbed as expected. However, since the rapid rise was not tracked equally by all participants – including, critically, some gas price Oracles – the average price to get mined began to diverge from the average price of transactions entering the mempool. And typical fee bumping was insufficient to keep up with the rate of price increases.
On March 12, our data platform detected significant separation of transaction fee pricing. Instead of the expected lag in offered gas price between some portion of the mempool and the marketable transactions – as seen with similar gas price changes on March 13 – pricing on March 12 between mined and un-mined transactions effectively bifurcated.
The typical behavior we observe is that unmarketable transactions are adjusted to become marketable, usually with a simple upward fee adjustment. However, on March 12 this was not the case. Instead, significant mempool resources were consumed by underpriced transactions resubmitted at nearly identical prices.
Box plot of gas prices each hour of March 12-13, 2020. For each hour, the box is bounded by the 1st and 3rd quartile values, a line inside shows the median, and the whiskers indicate minimum and maximum. Outliers are ignored (1.5 IQR).
Does this reflect organic behavior? Or might these underpriced transactions have been intended to flood the mempool? And if so, why?
Our forensic mempool archive data indicates that bots successfully increased congestion and distorted the gas price distribution in the mempool without any corresponding increase in transaction fees.
This had the net effect of increasing the stuck transaction rate and skewing gas price Oracles. This resulted in anomalous mempool conditions, which would ultimately favor certain transactions – namely, increasing the probability that a liquidated CDP position could be won at auction with a zero bid.
The bots hammered the mempool with transactions that were never intended to be finalized. These ‘Hammerbots’ consumed mempool resources by issuing extremely high rates of replacement transactions without any corresponding increase in gas. But how can transactions be replaced without the required minimum 10% fee increase designed to prevent precisely this sort of activity?
The simple answer: node amnesia caused by abnormally high rates of transaction eviction (see above).
The Hammerbots waited for – or simply anticipated – the eviction of their own transactions from the mempool and then immediately sent new replacement transactions with identical, or even reduced, underpriced gas fees. Having ‘forgotten’ the previously evicted transaction, nodes dutifully accepted the replacements as valid new transactions. Of course, this had the effect of further compounding congestion.
The Hammerbot transaction payloads themselves appear to be malformed in a distinctly ‘automated’ fashion, with each replacement including slightly modified contract inputs. Thus, each Hammerbot transaction had a unique hash and therefore could bypass the peer-to-peer network protocol spam filtering protections in each node.
As the pink overlays in the diagrams below highlight, starting at exactly 09:00 UTC on March 12, our mempool data platform detected a rapid increase in pending transactions that would never be mined. The rate of these then doubled approximately 35 minutes later, only to then drop off into a linear escalation pattern at approximately 10:00 UTC.
Per minute count of transactions entering the mempool from 08:00 UTC to 10:00 UTC on March 12. The blue line counts transactions that eventually finalized on-chain. The orange line counts transactions that are never finalized on chain.
Given that there appeared to be no intention of these transactions getting on-chain, the gas prices for these never-mined Hammerbot transactions remained abnormally low: almost always 5 Gwei when the market required ~30 Gwei or more. Hammerbot transactions efficiently consumed mempool resources without the normal ramping of gas price typical of speedup transactions used by arbitrage bots.
When viewed in aggregate, even though the volume of transactions entering the mempool increased dramatically, the gas price of a significant portion of the mempool collapsed to an artificially low value.
Box plot of gas prices each minute from 08:00 UTC to 10:00 UTC on March 12. The blue line counts transactions that eventually finalized on-chain. The orange line counts transactions that are never finalized on chain.
Exploring a sample Hammerbot transaction, in this case nonce 3070, is illustrative. Here is the final transaction hash:
Looking up this transaction in your favorite block explorer will not reveal much insight beyond its final confirmation on-chain. On the surface, this transaction appears unremarkable. But Blocknative's mempool data platform detected 418 unique replacement transactions preceding this nonce. This occurred in under an hour, with a replacement transaction every 6.86 seconds on average, with a gap of less than 0.1 seconds between some replacements.
Same nonce unique transaction count per minute across 4 nonces for address 0x5cf2fa4e0c84e71fd2e4fa86d2fa64b7a50a6fc0 that started at 09:00 UTC.
Most of these replacement transactions used an identical gas price, with the Hammerbot relying on mempool eviction mechanics to successfully enter each node's mempool as a 'new' transaction. Only the last replacement required a higher gas price in order to be marketable and therefore become finalized in a mined block.
Same nonce average gas price each minute across 4 nonces for address 0x5cf2fa4e0c84e71fd2e4fa86d2fa64b7a50a6fc0 that started at 09:00 UTC.
These patterns make it difficult to dismiss Hammerbot transaction behavior as accidental or unintentional. In fact, our systems recorded more than 20 unique candidate Hammerbot addresses active during the 60 second period starting at 9:05 UTC.
If you are the owner or operator of any of these candidate Hammerbot addresses, we would love to talk. Please get in touch.
Note: The characteristic Hammerbot behavior of resubmitting variants of evicted transactions at the same gas price may also be related to 'backrunning' actions recently described and discussed here and here.
MakerDAO’s collateralized debt positions (CDPs) allow anyone to borrow DAI by locking ETH into a smart contract. Your locked ETH acts as collateral for your DAI debt. Since ETH is a volatile asset, while DAI seeks to maintain a price of exactly $1.00, the amount of collateral required to maintain an open CDP position constantly changes.
When the price of ETH collapsed on March 12, a large number of CDPs immediately became undercollateralized and eligible for forcible liquidation. MakerDAO and the Ethereum ecosystem are incentivized to operate various Keeper bots in order to ensure a competitive marketplace for liquidated CDP positions. Such liquidations occur in auctions.
Any Keeper can participate in an auction with an on-chain bid, and any other Keeper can submit a higher counter-bid on-chain within 10 minutes. When a new bid is received, a fresh 10 minute bid window starts. If no higher bids are received within that 10 minute window, the auction is closed, and the last bid wins. [Note: In the aftermath of Black Thursday, MakerDAO increased the auction period from 10 minutes to 6 hours.]
However, when miner nodes were dropping transactions at an unusually high rate due to network congestion and mempool compression, many Keeper transactions were adversely impacted by stuck transactions and thus were delayed in successfully placing CDP liquidation auction counter-bids on-chain. The end result was that Keepers were unable to reliably compete in the time-windowed auctions initiated by a zero bid, even though Keeper bots correctly detected the zero bid auction start and responded with a much higher bid.
As a result of the congestion and fee confusion, Keepers failed to appropriately interpret the rapid gas price increases and Keeper transactions were thus evicted from miner mempools. This created the conditions for transaction nonce gaps to emerge in default configuration miner nodes. All subsequent bids for additional auctions by the Keeper bots then became stuck, essentially blocking the Keepers from participating in the 10-minute auction window and allowing the zero bid transactions to win out.
Here is an example sequence for auction #1866 illustrating the above behavior:
A 'zero bid bot' initiated the auction with a zero bid submitted at 15:59:50 UTC with a gas price of 200 Gwei. This transaction was confirmed and successfully went on-chain 26 seconds later. And the 10 minute bid countdown began.
A 'good faith Keeper bot' submitted a counter bid transaction at 16:08:01 with a gas price of 450 Gwei. This transaction was within the 10 minute bid window – 8 minutes and 11 seconds – and had 2.5x the gas price of the original zero bid transaction. If confirmed within the next one minute and 49 seconds, Auction #1866 would have proceeded as expected. However, this bid was effectively blocked by a nonce gap caused by an earlier evicted bid from the good faith Keeper bot.
The good faith Keeper bot sent a follow-up counter-bid with a 4,500 Gwei (!!) gas price at 16:15:31. The 10x gas price increase had no effect because the sending address remained stuck by the nonce gap created by the earlier evicted transaction. Furthermore, this counter-bid was sent outside the 10 minute auction window and would have failed once it became unstuck.
This pattern was repeated probabilistically throughout the day, resulting in $8.32 million in aggregated CDP holder losses. It is worth noting that much of the Hammerbot activity appears to have occurred earlier in the day, thus creating the congested conditions that later interfered with Keeper bot transactions.
The rapid increase in gas price, and the distortion of gas price estimation due to mempool compression, also negatively impacted other price Oracles. This caused further confusion with pricing and likely delayed re-collateralization attempts by CDP holders at risk of liquidation.
In summary, our forensic mempool investigation into the events of Black Thursday indicates the following:
Multiple Hammerbots saturated mining operation mempools. This increased and accelerated Ethereum network congestion.
Sending many same-gas-price replacement transactions created significant overhead in both the peer-to-peer gossip layer and the nodes themselves.
This impeded normal transaction propagation, causing a large number of transactions to be evicted or rejected from miner mempools.
This in turn led to an increased likelihood of creating a nonce gap and the associated stuck transactions. This also skewed gas price calculations.
While automated trading systems are often designed to programmatically increase the gas price of transactions, many such trading systems do not handle nonce gaps well – if at all.
The above are in-process findings. Our data set may yet reveal additional evidence and/or new findings that support or undermine our conclusions.
To accelerate the discovery process, we are now making a sample of our Mempool Archive data for those two days available to the community (see below). We hope this catalyzes further exploration into the role of the mempool in the events of Black Thursday and in other Ethereum network anomalies .
The mempool is a critical – yet ephemeral and often overlooked – element of the blockchain ecosystem. As such, mempools present many ‘unknown unknowns’ to builders and users alike.
At this stage, we do not know how often techniques like these are exploited in the wild – only that they appear to be actively exploited. And we do not know how many related exploits exist – only that sophisticated exploits appear to have been demonstrated to be effective in the real-world.
As a result, we recommend that all exchanges, protocols, wallet providers, and traders:
Constantly monitor mempool conditions, with a focus on 1) detecting when mempool congestion begins to build and 2) changes in transaction eviction and rejection rates.
Understand the nuances of transaction nonce ordering. And how even properly constructed transactions can become stuck in portions of the network.
Proactively watch for stuck transactions. And know which prior transaction(s) must be sped up in order to unblock a given address.
Calculate gas prices based on the marketable portion of the mempool. And be knowledgeable about the algorithms used by the gas price Oracles you rely on.
Do not assume predictable pending transaction behavior during periods of high congestion. Instead, monitor each and every transaction for the factors described here, including insufficient gas, dropped transactions, stuck transactions, front-running, and more.
These issues are exceptionally challenging, time-consuming, and expensive to address on your own. Blocknative's global data platform has been designed expressly to capture and normalize decentralized mempools – addressing each of the above recommendations along the way. Our infrastructure and APIs provide real-time mempool monitoring to keep your transactions flowing reliably, resiliently, and predictably.
If you are handling real value and want to start protecting yourself and your stakeholders from mempool manipulations like those described here, please contact us.
To help raise visibility into mempool mechanics in the real world, we are now publishing a sample of our forensic mempool archive describing Black Thursday. We invite the community to explore this data set and to collaboratively share and publish your findings.
You can download the data from our public S3 buckets here:
https://bnc-mainnet-2020-03-12.s3.amazonaws.com/readme.txt
https://bnc-mainnet-2020-03-13.s3.amazonaws.com/readme.txt
Each day has its own bucket. The data within each day is further divided into hourly files. See the readme.txt files for details on formats and interpretation. If you have questions about the data, please join the Blocknative Discord for support.
If you're looking to learn more about the pre-chain layer, we offer guides such as What is ETH Gas and What is Ethereum Transaction Simulation.
A special thanks to the many members of the ecosystem who reviewed prior drafts of this post, including Sarah Baker-Mills, Dmitriy Berenzon, Spencer Bogart, Nic Carter, Hsin-Ju Chuang, Tomasz Drwięga, Andy Gray, Hudson Jameson, Jon Kol, Calvin Liu, Justin Mart, Gavin McDermott, Taylor Monahan, Andra Nicolau, Charlie Noyes, Simona Pop, Alex Pruden, Austin Roberts, Cuy Sheffield, Larry Sukernik, Chris Whinfrey, and others. We deeply appreciate your feedback, insight, and guidance.
Mempool 'Compression' — a dramatic reduction in the marketable portion of the mempool (that is, transactions with sufficient gas to be considered by miners), which can bias ETH gas price estimates.
'Hammerbots' — Automated transaction systems that magnified congestion, and hence compounded mempool compression.
We are publishing this analysis and the associated data set in efforts to raise awareness of the dynamic transaction risks that can rapidly emerge during periods of Blockchain network congestion.
On March 12, 2020, the crypto markets saw one of their biggest sell-offs in history, with the price of ETH falling 43% – and BTC falling 39% – in a matter of hours. As prices fell, a negative feedback loop emerged that reduced liquidity and forced liquidations from various DeFi contracts. This downward pressure created real challenges for every entity attempting to transact on the network during this period.
Hourly weighted average ETH prices from March 12 through March 13, 2020 UTC. Source: cointelegraph.com.
The rapid price change caused sustained congestion in the Ethereum mempool, as automated transaction systems, such as trading bots, programmatically reacted to the volatility.
One negative consequence of this congestion were 'zero bid auctions' on liquidated MakerDAO CDPs. Of the 3,994 liquidation auctions associated with Black Thursday, 1,462 or 36.6% were won by zero bids. Over a roughly 12 hour period, $8.32 million in aggregated locked CDP value was lost to these zero bid auctions. More on this below and via MakerDAO's original The Market Collapse of March 12-13, 2020: How It Impacted MakerDAO post.
The rapid price change in ETH on March 12 created congestion in the mempool. This was exacerbated by price-triggered bot activity.
Hourly count of new pending transactions arriving into the mempool versus confirmed transaction count.
The result was a surge of transactions propagating through the network which forced nodes with default mempool configurations to actively protect their system resources by:
Evicting, or dropping, many valid transactions.
Rejecting, or ignoring, many valid transactions.
Mempools generally evict the transactions with the lowest associated fees. And although these transactions are not truly ‘lost’ across the entire network, evicted transactions may be significantly delayed until network conditions normalize.
Hourly count of evicted, or dropped, transactions along with new stuck (not processable) transactions arriving for March 11, 2020 through March 14, 2020.
Critically, evictions have a side effect of increasing the likelihood of stuck transactions.
When a transaction is evicted, the node does not remember any details about the transaction, such as the sending address or the transaction nonce. Therefore, when a new transaction arrives with the next nonce, the node may detect a nonce gap and will be unable to process the new transaction. Impacted transactions are then placed in the unprocessable Queued portion of the node's mempool. These transactions become stuck regardless of their associated transaction fees (that is, gas prices).
For more on this, please see our prior post covering transaction ordering: A Note On Nodes: Your Gateway To The Mempool.
Meanwhile, network congestion caused rapid increases in the minimum gas price needed to be admitted to a mempool, so the original transactions were now rejected as underpriced. And since the previously evicted transactions were unable to get back into the mempool, they could not span the transaction nonce gap. In fact, some node implementations actively ignore these rejected transactions for a while in efforts to prevent spamming over the peer-to-peer network. The nonce gap effectively locked addresses out of completing new transactions.
Stuck transactions were more likely to impact addresses that issue many new transactions, including automated trading systems, payment networks, and possibly even exchanges. These systems attempted to course correct, often compounding congestion as more and more transactions are delayed.
The only solution? You must first proactively detect when one of your transactions might be stuck. This can be challenging, as your transaction may only be stuck on some – but not all – nodes on the network. Then, you must identify the first missing transaction in the nonce gap and immediately speed up that transaction with a marketable gas price. Finally, you must continue to monitor the original stuck transaction to ensure that it has been moved from the Queued to the Pending portion of the mempool and has successfully proceeded to finalization. If your transaction remains stuck, repeat the process above until you can confirm that all transactions in the nonce gap have been successfully resolved. And yes, this is one of the many reasons why we built and operate our Notify API.
The increase in network congestion – and the resulting increase in stuck transactions – caused the marketable portion of the mempool to shrink. We call this 'mempool compression'.
Miners are incentivized to maximize revenue through block reward and gas utilization at the highest price. Mining operations make decisions on which block templates – that is, candidate blocks – to mine based on the offered gas prices of transactions in the mempool.
Due to congestion, several behaviors contributed to compressing the mempool so that a much smaller portion remained relevant to miners:
Exhausted resources: Depending on node implementation, the dramatic increase in stuck transactions consumed substantial mempool resources. This in turn reduced available node resources for processing valid pending transactions.
Escalating gas price: The falling ETH price triggered many 'high urgency' transactions that were structured to be confirmed before the value of ETH fell even lower. This drove both individual users and automated bots to rapidly escalate gas prices. Since not all participants were paying close attention to the congestion, the aggregate profile of pending mempool transactions included an unusually large number of underpriced – and hence unmarketable – entries.
Insufficiently responsive pricing algorithms: The gas price distribution of pending transactions in the mempool, and of those confirmed in recently mined blocks, could distort gas price Oracle calculations. This effectively underpriced gas for broad swaths of new transactions, further exacerbating pending transaction gas price disparities.
As mempool congestion rose, the gas price required to be mined and get on-chain climbed as expected. However, since the rapid rise was not tracked equally by all participants – including, critically, some gas price Oracles – the average price to get mined began to diverge from the average price of transactions entering the mempool. And typical fee bumping was insufficient to keep up with the rate of price increases.
On March 12, our data platform detected significant separation of transaction fee pricing. Instead of the expected lag in offered gas price between some portion of the mempool and the marketable transactions – as seen with similar gas price changes on March 13 – pricing on March 12 between mined and un-mined transactions effectively bifurcated.
The typical behavior we observe is that unmarketable transactions are adjusted to become marketable, usually with a simple upward fee adjustment. However, on March 12 this was not the case. Instead, significant mempool resources were consumed by underpriced transactions resubmitted at nearly identical prices.
Box plot of gas prices each hour of March 12-13, 2020. For each hour, the box is bounded by the 1st and 3rd quartile values, a line inside shows the median, and the whiskers indicate minimum and maximum. Outliers are ignored (1.5 IQR).
Does this reflect organic behavior? Or might these underpriced transactions have been intended to flood the mempool? And if so, why?
Our forensic mempool archive data indicates that bots successfully increased congestion and distorted the gas price distribution in the mempool without any corresponding increase in transaction fees.
This had the net effect of increasing the stuck transaction rate and skewing gas price Oracles. This resulted in anomalous mempool conditions, which would ultimately favor certain transactions – namely, increasing the probability that a liquidated CDP position could be won at auction with a zero bid.
The bots hammered the mempool with transactions that were never intended to be finalized. These ‘Hammerbots’ consumed mempool resources by issuing extremely high rates of replacement transactions without any corresponding increase in gas. But how can transactions be replaced without the required minimum 10% fee increase designed to prevent precisely this sort of activity?
The simple answer: node amnesia caused by abnormally high rates of transaction eviction (see above).
The Hammerbots waited for – or simply anticipated – the eviction of their own transactions from the mempool and then immediately sent new replacement transactions with identical, or even reduced, underpriced gas fees. Having ‘forgotten’ the previously evicted transaction, nodes dutifully accepted the replacements as valid new transactions. Of course, this had the effect of further compounding congestion.
The Hammerbot transaction payloads themselves appear to be malformed in a distinctly ‘automated’ fashion, with each replacement including slightly modified contract inputs. Thus, each Hammerbot transaction had a unique hash and therefore could bypass the peer-to-peer network protocol spam filtering protections in each node.
As the pink overlays in the diagrams below highlight, starting at exactly 09:00 UTC on March 12, our mempool data platform detected a rapid increase in pending transactions that would never be mined. The rate of these then doubled approximately 35 minutes later, only to then drop off into a linear escalation pattern at approximately 10:00 UTC.
Per minute count of transactions entering the mempool from 08:00 UTC to 10:00 UTC on March 12. The blue line counts transactions that eventually finalized on-chain. The orange line counts transactions that are never finalized on chain.
Given that there appeared to be no intention of these transactions getting on-chain, the gas prices for these never-mined Hammerbot transactions remained abnormally low: almost always 5 Gwei when the market required ~30 Gwei or more. Hammerbot transactions efficiently consumed mempool resources without the normal ramping of gas price typical of speedup transactions used by arbitrage bots.
When viewed in aggregate, even though the volume of transactions entering the mempool increased dramatically, the gas price of a significant portion of the mempool collapsed to an artificially low value.
Box plot of gas prices each minute from 08:00 UTC to 10:00 UTC on March 12. The blue line counts transactions that eventually finalized on-chain. The orange line counts transactions that are never finalized on chain.
Exploring a sample Hammerbot transaction, in this case nonce 3070, is illustrative. Here is the final transaction hash:
Looking up this transaction in your favorite block explorer will not reveal much insight beyond its final confirmation on-chain. On the surface, this transaction appears unremarkable. But Blocknative's mempool data platform detected 418 unique replacement transactions preceding this nonce. This occurred in under an hour, with a replacement transaction every 6.86 seconds on average, with a gap of less than 0.1 seconds between some replacements.
Same nonce unique transaction count per minute across 4 nonces for address 0x5cf2fa4e0c84e71fd2e4fa86d2fa64b7a50a6fc0 that started at 09:00 UTC.
Most of these replacement transactions used an identical gas price, with the Hammerbot relying on mempool eviction mechanics to successfully enter each node's mempool as a 'new' transaction. Only the last replacement required a higher gas price in order to be marketable and therefore become finalized in a mined block.
Same nonce average gas price each minute across 4 nonces for address 0x5cf2fa4e0c84e71fd2e4fa86d2fa64b7a50a6fc0 that started at 09:00 UTC.
These patterns make it difficult to dismiss Hammerbot transaction behavior as accidental or unintentional. In fact, our systems recorded more than 20 unique candidate Hammerbot addresses active during the 60 second period starting at 9:05 UTC.
If you are the owner or operator of any of these candidate Hammerbot addresses, we would love to talk. Please get in touch.
Note: The characteristic Hammerbot behavior of resubmitting variants of evicted transactions at the same gas price may also be related to 'backrunning' actions recently described and discussed here and here.
MakerDAO’s collateralized debt positions (CDPs) allow anyone to borrow DAI by locking ETH into a smart contract. Your locked ETH acts as collateral for your DAI debt. Since ETH is a volatile asset, while DAI seeks to maintain a price of exactly $1.00, the amount of collateral required to maintain an open CDP position constantly changes.
When the price of ETH collapsed on March 12, a large number of CDPs immediately became undercollateralized and eligible for forcible liquidation. MakerDAO and the Ethereum ecosystem are incentivized to operate various Keeper bots in order to ensure a competitive marketplace for liquidated CDP positions. Such liquidations occur in auctions.
Any Keeper can participate in an auction with an on-chain bid, and any other Keeper can submit a higher counter-bid on-chain within 10 minutes. When a new bid is received, a fresh 10 minute bid window starts. If no higher bids are received within that 10 minute window, the auction is closed, and the last bid wins. [Note: In the aftermath of Black Thursday, MakerDAO increased the auction period from 10 minutes to 6 hours.]
However, when miner nodes were dropping transactions at an unusually high rate due to network congestion and mempool compression, many Keeper transactions were adversely impacted by stuck transactions and thus were delayed in successfully placing CDP liquidation auction counter-bids on-chain. The end result was that Keepers were unable to reliably compete in the time-windowed auctions initiated by a zero bid, even though Keeper bots correctly detected the zero bid auction start and responded with a much higher bid.
As a result of the congestion and fee confusion, Keepers failed to appropriately interpret the rapid gas price increases and Keeper transactions were thus evicted from miner mempools. This created the conditions for transaction nonce gaps to emerge in default configuration miner nodes. All subsequent bids for additional auctions by the Keeper bots then became stuck, essentially blocking the Keepers from participating in the 10-minute auction window and allowing the zero bid transactions to win out.
Here is an example sequence for auction #1866 illustrating the above behavior:
A 'zero bid bot' initiated the auction with a zero bid submitted at 15:59:50 UTC with a gas price of 200 Gwei. This transaction was confirmed and successfully went on-chain 26 seconds later. And the 10 minute bid countdown began.
A 'good faith Keeper bot' submitted a counter bid transaction at 16:08:01 with a gas price of 450 Gwei. This transaction was within the 10 minute bid window – 8 minutes and 11 seconds – and had 2.5x the gas price of the original zero bid transaction. If confirmed within the next one minute and 49 seconds, Auction #1866 would have proceeded as expected. However, this bid was effectively blocked by a nonce gap caused by an earlier evicted bid from the good faith Keeper bot.
The good faith Keeper bot sent a follow-up counter-bid with a 4,500 Gwei (!!) gas price at 16:15:31. The 10x gas price increase had no effect because the sending address remained stuck by the nonce gap created by the earlier evicted transaction. Furthermore, this counter-bid was sent outside the 10 minute auction window and would have failed once it became unstuck.
This pattern was repeated probabilistically throughout the day, resulting in $8.32 million in aggregated CDP holder losses. It is worth noting that much of the Hammerbot activity appears to have occurred earlier in the day, thus creating the congested conditions that later interfered with Keeper bot transactions.
The rapid increase in gas price, and the distortion of gas price estimation due to mempool compression, also negatively impacted other price Oracles. This caused further confusion with pricing and likely delayed re-collateralization attempts by CDP holders at risk of liquidation.
In summary, our forensic mempool investigation into the events of Black Thursday indicates the following:
Multiple Hammerbots saturated mining operation mempools. This increased and accelerated Ethereum network congestion.
Sending many same-gas-price replacement transactions created significant overhead in both the peer-to-peer gossip layer and the nodes themselves.
This impeded normal transaction propagation, causing a large number of transactions to be evicted or rejected from miner mempools.
This in turn led to an increased likelihood of creating a nonce gap and the associated stuck transactions. This also skewed gas price calculations.
While automated trading systems are often designed to programmatically increase the gas price of transactions, many such trading systems do not handle nonce gaps well – if at all.
The above are in-process findings. Our data set may yet reveal additional evidence and/or new findings that support or undermine our conclusions.
To accelerate the discovery process, we are now making a sample of our Mempool Archive data for those two days available to the community (see below). We hope this catalyzes further exploration into the role of the mempool in the events of Black Thursday and in other Ethereum network anomalies .
The mempool is a critical – yet ephemeral and often overlooked – element of the blockchain ecosystem. As such, mempools present many ‘unknown unknowns’ to builders and users alike.
At this stage, we do not know how often techniques like these are exploited in the wild – only that they appear to be actively exploited. And we do not know how many related exploits exist – only that sophisticated exploits appear to have been demonstrated to be effective in the real-world.
As a result, we recommend that all exchanges, protocols, wallet providers, and traders:
Constantly monitor mempool conditions, with a focus on 1) detecting when mempool congestion begins to build and 2) changes in transaction eviction and rejection rates.
Understand the nuances of transaction nonce ordering. And how even properly constructed transactions can become stuck in portions of the network.
Proactively watch for stuck transactions. And know which prior transaction(s) must be sped up in order to unblock a given address.
Calculate gas prices based on the marketable portion of the mempool. And be knowledgeable about the algorithms used by the gas price Oracles you rely on.
Do not assume predictable pending transaction behavior during periods of high congestion. Instead, monitor each and every transaction for the factors described here, including insufficient gas, dropped transactions, stuck transactions, front-running, and more.
These issues are exceptionally challenging, time-consuming, and expensive to address on your own. Blocknative's global data platform has been designed expressly to capture and normalize decentralized mempools – addressing each of the above recommendations along the way. Our infrastructure and APIs provide real-time mempool monitoring to keep your transactions flowing reliably, resiliently, and predictably.
If you are handling real value and want to start protecting yourself and your stakeholders from mempool manipulations like those described here, please contact us.
To help raise visibility into mempool mechanics in the real world, we are now publishing a sample of our forensic mempool archive describing Black Thursday. We invite the community to explore this data set and to collaboratively share and publish your findings.
You can download the data from our public S3 buckets here:
https://bnc-mainnet-2020-03-12.s3.amazonaws.com/readme.txt
https://bnc-mainnet-2020-03-13.s3.amazonaws.com/readme.txt
Each day has its own bucket. The data within each day is further divided into hourly files. See the readme.txt files for details on formats and interpretation. If you have questions about the data, please join the Blocknative Discord for support.
If you're looking to learn more about the pre-chain layer, we offer guides such as What is ETH Gas and What is Ethereum Transaction Simulation.
A special thanks to the many members of the ecosystem who reviewed prior drafts of this post, including Sarah Baker-Mills, Dmitriy Berenzon, Spencer Bogart, Nic Carter, Hsin-Ju Chuang, Tomasz Drwięga, Andy Gray, Hudson Jameson, Jon Kol, Calvin Liu, Justin Mart, Gavin McDermott, Taylor Monahan, Andra Nicolau, Charlie Noyes, Simona Pop, Alex Pruden, Austin Roberts, Cuy Sheffield, Larry Sukernik, Chris Whinfrey, and others. We deeply appreciate your feedback, insight, and guidance.
No comments yet