The Shwap upgrade: 12x faster DA sampling
Celestia Mainnet Beta recently completed its first upgrade, introducing the first new set of features to the consensus network.
The data availability (DA) network is scheduled to receive its first upgrade with Shwap - now activated on Arabica and Mocha testnets with v0.18.2 of celestia-node. Shwap makes DA sampling 12x faster and reduces storage requirements by 16.5x, enabling bigger blocks and smaller nodes.
Shwap is another step towards the Celestia community roadmap’s core objective: relentlessly scale to 1 gigabyte blocks. Specified in CIP-19, Shwap includes a new messaging protocol and storage system for the DA network.
Shwap enables DA throughput to massively scale while keeping light node requirements low. High throughput DA accelerates existing crypto apps and enables new, unstoppable applications using any VM. And lower requirements enable anyone to run an in-browser or in-wallet light node, paving the way for truly verifiable web apps.
Following further testing, Shwap is expected to reach Mainnet Beta during November. Keep in mind that the Celestia community will drive the precise timing of network upgrades.
The DA network as it stands
The DA network can be described as two protocols. One is responsible for circulating chain headers via go-header, and the other manages the actual block data in the form of a data square consisting of little shares or samples. These shares are sampled by light nodes. And the current protocol and storage system are designed around the sampling protocol introduced during the good old Devnet era. This IPLD-based protocol has proven functional but is fundamentally unscalable, especially with the north star of 1GB blocks.
The protocol and its storage followed the typical pitfall in blockchain engineering: hash-addressability. It sounds like a simple and powerful concept providing “nice” features like deduplication. It is general enough to apply to any hash-based data structure like state storage or, in our case, data square storage. Altogether, hash addressability feels like a simple goto solution, and indeed, it bootstrapped our engineering with all the IPLD tooling out there; however, as always, the devil is in the details.
For instance, a light node must download the share and its associated Merkle proof to verify a sample. Under the hash-based system, each node in the proof is retrieved separately. For a square size of 128x128, this process requires the light node to make 7 blocking round-trips, significantly slowing sampling. This was the case for the old sampling protocol and storage. A light node had to perform 7 network requests followed by 7 IO reads on the serving full node side. Beyond that, the server full node had to maintain a global index of historical Merkle Tree nodes, staggering overall data throughput due to key-value store compactions.
To conclude, the path of hash addressability led to:
- Storing Merkle proofs instead of recomputing them on the fly.
- Sticking to O(log2n) data access pattern for something that can be O(1).
These decisions do not allow the old protocol to scale, requiring a fundamentally different approach and starting afresh, leading to Shwap.
Shwap
The Shwap (share + swap) upgrade introduces a new messaging framework and storage system that enhances sampling and data efficiency retrieval. It reduces full node storage requirements by 16.5x and provides a scalable foundation for ever-growing block sizes.
The main changes Shwap introduces are:
- New data square storage subsystem
- Composable networking framework
- O(1) data availability sampling (DAS)
In Shwap, the previous inefficient and experimental version of block reconstruction is also removed in favor of a more efficient production-ready version in a future upgrade. Dive into Shwap by watching the Modular Summit talk here.
Square Storage
The data square storage subsystem has been reworked to avoid storing the historical Merkle proofs and the global index. It also eliminates unnecessary metadata stored with the initially chosen IPLD-based storage dependency. Overall, the new data square storage architecture yields a staggering 16.5x improvement compared to the old storage usage 1:1.
Our benchmarks show that recomputing hashes with modern CPUs takes less time than reading them from NVMes. This renders storing all proofs effectively useless, albeit storing a few strategically.
Additionally, the new storage introduces a new immutable file format that supports storing optional parity data, accelerating sampling retrieval by minimizing IO and compute.
Networking
Shwap provides a flexible and extensible messaging framework for Celestia's DA network. It standardizes messaging for share exchanges without enforcing specific transport protocols, supporting both TCP/QUIC and application layer protocols like HTTP/x. The key networking improvements include:
Expressivity
Shwap introduces data Containers that enable granular access to any part of the data square. It can be Sample, Share Range, Blob, Row, NamespaceData, etc. At its core Shwap brings expressivity to the data square, allowing anyone to efficiently retrieve posted data in any form grouped within a Container. This powerful feature enables the community to extend the capabilities of the data square yet makes sure that software quickly adapts the new features.
Protocol decoupling
Shwap decouples core networking messages from underlying transport and protocol layers, allowing it to be stacked with lower-level protocols such as Bitswap, KadDHT, ShrEx, or custom protocols.
ShrEx integration
Usage of Shwap over ShrEx protocol improves block data synchronization, accelerating data retrieval and reducing resource consumption. With Shwap’s efficient data storage, ShrEx makes sync speeds 22x faster, reducing sync time for full nodes on Mocha from 7 days to 8 hours.
Integration with Bitswap for efficient DAS
Shwap dramatically improves data availability sampling (DAS) by integrating with Bitswap. Previously, DAS required 7 round-trips to retrieve data for a 128x128 data square, but with Shwap’s messaging framework, Bitswap now handles this in just 1 round-trip.
These networking improvements form the backbone of the Shwap upgrade, enabling more efficient communication in Celestia’s DA network and providing significant performance gains across the board.
Light node
The Shwap upgrade enhances light nodes, particularly in terms of DAS efficiency. With Shwap’s O(1) round-trip sampling, light nodes can perform DAS 12x faster and reduce sync time from 24 hours to just 2 hours.
Additionally, Shwap optimizes the way samples are stored, reducing the storage footprint for light nodes by 2.66x, making light nodes more efficient while maintaining the same level of network participation.
Future Endeavors
This is a significant step towards the community’s goal of 1GB block data squares; however, there’s still work to do. Shwap lays the foundation for the software architecture needed to get there, yet many more optimizations must be incorporated to scale to that target.
Now, with the proper foundation, the DA network is not expected to see significant storage and networking protocol rewrites going forward, enabling core devs to focus on maintenance, strengthening security, horizontal feature additions, and, most importantly, squeezing more performance.