Ethereum’s decentralized mantra in tatters after execution of client bug

Getting your Trinity Audio player ready...

Ethereum’s claims of decentralization are ringing ever more hollow due to the network’s lack of client software diversity, an overreliance that could pose an existential threat to ETH stakers.

Last weekend, around 8% of Ethereum proof-of-stake (PoS) transaction validators suddenly began producing invalid blocks due to a critical flaw in the Nethermind client software. The issue followed the release of Netherland’s v1.23.0 update, requiring a frantic patch to get these network nodes back in business.

While the order was eventually restored, a similar scenario befell the Besu client earlier this month. Besu’s share of the Ethereum execution client market was around 5% at the time and has since fallen to 4%.

Ethereum watchers soon warned that the fallout could be catastrophic if/when a similar situation impacted the Geth (Go Ethereum) client. Geth accounted for around 84% of network execution clients at the time of the Nethermind bug, but a concerted effort to sound the ‘eggs in one basket’ alarm has since pushed this down to a mere 79%.

Geth is client software developed by the Ethereum Foundation that supports network functions like transaction validation and smart contract execution. Geth is generally considered a more robust option than its rivals, which, along with the Foundation’s stamp of approval, is why it accounts for such an outsized market share.

Regardless, such a high concentration level would be problematic in any sector. But in an industry so prone to probing by malicious actors—particularly those adept at software supply chain attacks—it’s a recipe for disaster.

A Geth failure would bring Ethereum finalization to a screeching halt, but as Labrys developer Lachlan Feeney recently suggested, this vulnerability could also result in the loss of the majority of the roughly 29 million ETH currently staked by validators.

Because validators are penalized for being offline, an ‘inactivity leak’ could rob validators of two months’ worth of staking rewards in just two days. Should the downtime extend for five days, a year’s worth of rewards would be lost. A weeklong outage could cost validators 10% of their staked ETH, while 90% of that stake could be lost if the outage extends to six weeks. While such a prolonged outage may be unlikely, it’s also not impossible.

Geth-based validators would likely stampede for the exits rather than watch their stake bleed away to nothing. But they’d get caught in a logjam of similarly minded validators, all of whom would continue bleeding ETH as they wait their turn to disembark this sinking ship. Given the sheer number of Geth-based validators, this bottleneck could mean that only one in 12 could exit with more than 50% of their stake.

Fork me

The potential repercussions of a serious Geth bug don’t stop there. Should a Geth-based validator produce an invalid block, Geth’s domination of the network could result in this invalid block being added to the chain, resulting in a fork that would quickly become the dominant chain. Geth validators would be blocked from the valid chain until the smaller chain is finalized.

As Feeney put it, “Because the Geth validators are stuck on the invalid chain, they are considered inactive on the non-Geth chain and will suffer the inactivity leak. No software update or bug patch to Geth will save these validators. They will be bled out until their stake represents <⅓ of the network, allowing the non-Geth chain to finalize.”

Feeney estimates that this bleeding could result in an 18% reduction in the total supply of ETH. (That’s definitely one method of creating artificial scarcity and thus losing the token’s fiat value.)

Feeney offered this warning: “Staked ETH is not risk-free yield. Would you invest a minimum of $75,000 USD [the rough value of the 32 ETH required for serving as a validator] into an instrument where the maximum potential gain is 3.5% p.a. but the potential for loss is 100% (even if that loss is unlikely)? Probably not, but this is what 84% of the Ethereum stakers are doing today.”

Lido shuffle

Retail users lacking the 32 ETH necessary to stake on their own have several options for pooling their resources, but staking via a service won’t necessarily protect them from the potential carnage described above.

Lido Finance, the largest staking service with around 9.4 million ETH staked, relies on Geth for most of its operations. On January 23, Lido stated that its preliminary Q4/23 data put Geth usage across Lido protocol validators at 67%, down from 76% in Q3/23 and 93% in Q3/22. Lido said “client diversity is fundamental” to its “mission to decentralize Ethereum.”

Lido added that its Lido DAO (decentralized autonomous organization) node operators “are afforded high degrees of autonomy,” but they “have already begun to signal their commitments to reduce majority client usage, or explain how their setups avoid the possibility of being affected by supermajority bugs.”

Coinbase has a plan to make a plan

The Coinbase (NASDAQ: COIN) exchange has faced similar queries this week about its reliance on Geth. On January 22, CEO Brian Armstrong personally responded to a Coinbase customer who tweeted that they had “unstaked all of the ETH I had staked with you since you offered it as a service.” The user added that this “single client staking setup” made it “not worth the risk” of losing “a large percentage of my deposit.” Armstrong replied: “Taking a look.”

Later that day, the Coinbase Cloud account tweeted that when it launched its ETH staking service, “Geth was the only client that met our technical requirements.” Coinbase claimed that “execution client diversity … is a critical concern,” and thus, it was conducting “an updated technical assessment with the goal of adding another execution client to our infrastructure.” Coinbase promised to provide an update on its progress “by the end of February.”

Not everyone found this reassuring, with at least one customer suggesting that the situation wasn’t “a review and plan kind of phase. This is take serious and urgent action, with informed customers phase.” Coinbase was urged to set up “an insurance fund or allow us to opt in to other clients” because “the risk of supermajority failure is a *bigger* risk to your customers than minority client failure.”

This probably wasn’t the best week for Coinbase to run a sponsored post on Decrypt promoting the claim that its staking service “aims to be a ‘one-stop shop’ for crypto staking.” One stop, one point of failure… Synergy!

Coinbase rival exchanges Binance, Kraken, and Bitfinex (among others) also rely solely on Geth to power their staking services. However, they have kept quiet regarding any plans they may have to inject a little diversity into their operations.

All in one

Decentralization theater has been Ethereum’s stock-in-trade from its inception, starting with the Ethereum Foundation’s oversight-bereft crowd sale that delivered the majority of ETH into the hands of a few whales. This concentration of wealth and power continues to this day via the PoS consensus mechanism that further enriches the whales who can afford to run multiple validators.

This centralization was recently cited by the U.S. Securities and Exchange Commission (SEC) as part of the reason it was delaying decisions on applications to offer Ethereum spot-based exchange-traded funds (ETF). BlackRock, Grayscale Investments, and Fidelity are among those chomping at the bit to offer Ethereum ETFs to the public following the launch of multiple BTC spot-based ETFs earlier this month.

In approving those BTC ETFs, SEC chairman Gary Gensler stressed that the decision was limited to “one non-security commodity” (BTC) and “should in no way signal the Commission’s willingness to approve listing standards for crypto asset securities.” Gensler, who has previously stated his belief that ETH is an unregistered security, reiterated his ‘don’t expect ETH ETF approvals anytime soon’ message earlier this week.

On Thursday, the SEC posted a list of ETH-related questions for public comment before making any ETF decision. For instance, the SEC wonders if there are “particular features related to ETH and its ecosystem, including its proof of stake consensus mechanism and concentration of control or influence by a few individuals or entities, that raise unique concerns about ETH’s susceptibility to fraud and manipulation?”

Interested parties are instructed to submit their comments within three weeks. Just remember Ethereum Foundation members, you have to at least try to make it look like they’re not all coming from the same address.

Follow CoinGeek’s Crypto Crime Cartel series, which delves into the stream of groups—from BitMEX to Binance, Bitcoin.com, Blockstream, ShapeShift, Coinbase, Ripple, Ethereum,
FTX and Tether—who have co-opted the digital asset revolution and turned the industry into a minefield for naïve (and even experienced) players in the market.