Recently, an airdrop on the EOS mainnet has caused transaction rates on the network to skyrocket, reaching as much as 30x of the previous baseline. Many of the infrastructure services that EOS dapps depend on have struggled to keep up with this increase, or have completely crashed and remained down. dfuse has also felt the impact of this onslaught of traffic, but we’ve remained up and running, and are processing transactions as fast as the network can throw them at us -- up to an incredible 500 token transfers per second in recent days.
In this post we’ll cover what happened, and the challenges the dfuse team faced with storage, query planning and indexing, and abuse management as EOS traffic surged.
The EIDOS Airdrop
Since Nov 1, traffic levels on the EOS mainnet have grown from an average of 5 token transfers per second to a sustained level of over 500, with sudden peaks of close to a thousand. This was due to the launch of a unique style of airdrop for the EIDOS token. This airdrop makes use of smart contract named
eidosonecoin, which users are encouraged to send EOS tokens to. Any account that sends EOS tokens to the EIDOS smart contract receives their EOS back, plus a small number of EIDOS tokens. As only a tiny amount of EOS has to be sent (0.0001 EOS) in order to receive some EIDOS, and the resulting EIDOS tokens can be sold on several exchanges, large numbers of users have been sending repeated transactions to the EIDOS contract.
In addition to a dramatic increase in the transaction rate on the network, users seeking to interact repeatedly with the EIDOS contract have been packing many actions into each transaction, further increasing network activity levels.
Daily actions by contract::action - from labs.eostitan.com
EOS traffic surged as high as 1000 token transfers per second on November 11. This was brought about when one malicious actor took advantage of the unlimited CPU resource available to the
eosio system account, exploited through a bug in the
bidrefund actions that are part of the namespace auction on the EOS platform. This allowed the user to mine the EIDOS token without expending any of his own CPU resource, oversaturating the network beyond the defined limits set by Block Producers.
This bug was corrected on November 13 after a patch was released by Block.one charging the CPU resource to the account calling
bidname. Traffic has since leveled off at 500 per second, and seems likely to continue at or near that level for the remaining 14 months of the EIDOS airdrop, or until there is potentially no longer any interest in the EIDOS token due to a drop in price.
CPU usage by account - Provided by EOS Rio through Hyperion
Shows how much resources the attacker was using, as seen in red as the
This “new normal” represents several megabytes of data per block (compared with 100-150KB before the EIDOS airdrop began -- a 10-15x increase). With EOS’s fast interblock time of 0.5 seconds, this means that the EOS mainnet now transmits about 25GB of blockchain data over the network each day. Ingesting that volume of data daily, and integrating it with an already massive dataset, is obviously a huge challenge for a service like dfuse.
Impact of the Airdrop
The activity resulting from the EIDOS airdrop has resulted in the EOS network entering and remaining for an extended period of time in congestion mode. This means that all accounts are limited to their pro-rata share of total CPU resources. This directly affects the number of transactions any account will be able to perform per unit of EOS staked.
Average CPU utilization between October 18 and November 14 - from labs.eostitan.com
Additionally, this led many dapps to turn to REX to borrow CPU resources, with demand outstripping supply. Currently, REX cannot be utilized to borrow more resources, as it has passed a safety threshold for liquidity and is in a “liquidity crunch”.
REX usage as shown through fees paid into REX - from eosauthority.com
This has had an impact on dapp usage -- the number of active users on the EOS network has fallen, as many users find they lack the CPU resources to carry out network transactions.
It’s important to note that most dapps need not only to carry out transactions on the chain, but also to query or search the chain for transactions of interest. Since native EOS nodes are quite limited in their search capabilities, this is usually done with the help of an API service, such as dfuse and its competitors. However, most of these services have been unable to sustain the “new normal” traffic levels, with some having shut down completely.
Challenges and Solutions at dfuse
On November 1, our systems at dfuse were operating at load levels consistent with the old normal. In our case, because dfuse is built on a proprietary storage and query layer (dfuse DB), rather than simplistically load-balancing requests across many native blockchain nodes, that meant we had a large buffer of unused capacity. As a result, our systems were able to handle the initial 5x traffic surge without hitting saturation and the resulting classic cascade of failures.
However, alarms went off making our team aware of the traffic surge, and making them aware that our spare capacity had fallen below the safety thresholds we had defined.
Initial investigation showed that not only had incoming data surged by over 5x, but outgoing traffic -- dfuse’s responses to users’ queries and searches -- had also grown by a similar ratio, as many users had queries that returned data about large numbers of the now more numerous transactions on the network.
Our team immediately began reaching out to customers to alert them that their queries were generating more responses, which would result in increased costs, and worked with customers in some cases to fine tune their queries. Other customers needed all the data, as they needed all
Growth in documents streamed out by dfuse during this period
Fortunately, our core development team had recently shipped a set of internal improvements that re-engineered a core data structure in dfusedb to make it over 4x more space efficient, without increasing search time. This helped us cope with the massive increase in data size that the new traffic levels represented.
However, as our users demand lightning-fast responses to their searches and queries, we were concerned that growing query sizes could impact performance. Our core engineering team focused on this problem, and was able to find ways to further improve the blockchain-aware query planner component of dfuse DB.
This resulted in a more distributed indexing strategy that maintained an undiminished performance level, even in the face of massively larger data sets. This was not just a matter of spinning more nodes, but rather of intelligently distributing search and indexing tasks. As a result, our typically sub-second performance on full-history blockchain queries wasn’t impacted.
Finally, as the blockchain data set increased in size, resulting in some “free tier” users potentially exhausting their document quotas more quickly due to larger response sizes, we saw a significant increase in abuse rates. Our team detected some users creating new throwaway accounts daily, only to immediately use up the free quota of each. As a result, our team worked quickly to improve our usage quota management, to more fairly allocate resources between free and paid users.
How to Keep Your Dapp Alive
When selecting APIs and infrastructure services for your dapp, it pays to select services that can sustain complete accuracy and high performance even in adverse network conditions as well as providing you with the most powerful and performant capabilities, operated by recognized industry leaders, and supported by a passionate and dedicated team focused on delivering value to our customers and to the ecosystem as a whole.
dfuse provides you with a modern infrastructure layer for your dapps that is:
- Gives you highly granular, streaming access to blockchain events,
- Supports active webhook-style callbacks,
- all with the highest reliability in the industry.
Try dfuse today by signing up for your free account. Reach out to us on Twitter or Telegram with any questions / suggestions. We’d love to understand your data needs and how we can help your dapp to succeed.