The Development History and Future Prospects of Decentralization Storage

Storage has been one of the popular tracks in the blockchain industry. Filecoin, as the leading project of the previous bull market, once had a market value exceeding 10 billion USD. Arweave focuses on permanent storage, with a peak market value of 3.5 billion USD. However, as the practicality of cold data storage is questioned, the prospects of Decentralization storage are also being put into doubt.

Recently, the emergence of Walrus has brought new attention to the long-dormant storage narrative. The Shelby project, launched in collaboration by Aptos and Jump Crypto, aims to elevate decentralized storage in the hot data field to new heights. So, can decentralized storage make a comeback and become a widely adopted infrastructure? Or is it merely another round of speculation? This article will analyze the evolution of the decentralized storage narrative from the development paths of four projects: Filecoin, Arweave, Walrus, and Shelby, and explore its future development direction.

Filecoin: Storage is just a facade, mining is the essence

Filecoin is one of the early emerging cryptocurrency projects, and its development direction revolves around Decentralization. This is a common characteristic of early crypto projects - seeking the meaning of Decentralization in various traditional fields. Filecoin attempts to combine storage with Decentralization, proposing the viewpoint that centralized data storage carries trust risks. However, certain trade-offs made to achieve Decentralization have become pain points that later projects like Arweave or Walrus attempt to address.

To understand why Filecoin is essentially just a mining coin, it is necessary to comprehend the objective limitations of its underlying technology, IPFS, which is not suitable for hot data storage.

IPFS: Decentralization architecture, but limited by transmission bottlenecks.

IPFS( InterPlanetary File System) was introduced around 2015, aiming to disrupt the traditional HTTP protocol through content addressing. However, the biggest problem with IPFS is its extremely slow retrieval speed. In an era where traditional data service providers can achieve millisecond-level response times, it still takes IPFS several seconds to retrieve a file, making it difficult to promote in practical applications and explaining why it is rarely adopted by traditional industries, except for a few blockchain projects.

The underlying P2P protocol of IPFS is mainly suitable for "cold data" - static content that does not change frequently, such as videos, images, and documents. However, when it comes to handling hot data, such as dynamic web pages, online games, or AI applications, the P2P protocol does not have a significant advantage over traditional CDNs.

Although IPFS itself is not a blockchain, its directed acyclic graph (DAG) design concept is highly compatible with many public chains and Web3 protocols, making it inherently suitable as a foundational framework for blockchain. Therefore, even in the absence of practical value, it is sufficient as a foundational framework for carrying blockchain narratives. Early copycat projects only need a functional framework to launch grand visions, but when Filecoin reaches a certain stage of development, the limitations brought by IPFS begin to hinder its further progress.

Logic of mining coins under the storage cloak

The original intention of IPFS design is to allow users to not only store data but also be part of the storage network. However, without economic incentives, it is difficult for users to voluntarily use this system, let alone become active storage nodes. This means that most users will only store files on IPFS but will not contribute their own storage space or store others' files. It is against this backdrop that Filecoin emerged.

In the token economic model of Filecoin, there are mainly three roles: users are responsible for paying fees to store data; storage miners earn token incentives for storing user data; retrieval miners provide data when users need it and receive incentives.

This model has potential malicious exploitation space. Storage miners may fill garbage data after providing storage space in order to obtain rewards. Since this garbage data will not be retrieved, even if lost, it will not trigger the penalty mechanism for storage miners. This allows storage miners to delete garbage data and repeat this process. Filecoin's proof of replication consensus can only ensure that user data has not been deleted privately, but it cannot prevent miners from filling garbage data.

The operation of Filecoin largely relies on miners' continuous investment in the token economy, rather than on the actual demand for distributed storage from end users. Although the project is still iterating, at this stage, the ecological construction of Filecoin aligns more with the "mining coin logic" rather than the definition of a "application-driven" storage project.

Arweave: Born out of long-termism, defeated by long-termism

If Filecoin's design goal is to build a incentivized, verifiable decentralized "data cloud" shell, then Arweave takes a different extreme direction in storage: providing the ability for permanent storage of data. Arweave does not attempt to build a distributed computing platform; its entire system revolves around a core assumption - important data should be stored once and remain on the network forever. This extreme long-termism means that Arweave's mechanisms, incentive models, hardware requirements, and narrative perspectives are vastly different from those of Filecoin.

Arweave takes Bitcoin as its learning object, attempting to continuously optimize its permanent storage network over long periods measured in years. Arweave does not care about marketing, nor about competitors and market development trends. It is only focused on iterating the network architecture, moving forward even if no one pays attention, because this is the essence of the Arweave development team: long-termism. Thanks to long-termism, Arweave was highly sought after in the last bull market; and also because of long-termism, even after falling to the bottom, Arweave may still withstand several rounds of bull and bear markets. The only question is whether there will be a place for Arweave in the future of Decentralization storage? The value of permanent storage can only be proven through time.

Since version 1.5, the Arweave mainnet has been working hard to allow a broader range of miners to participate in the network with minimal costs, despite losing its market discussion momentum until the recent version 2.9. It incentivizes miners to maximize data storage, continuously enhancing the robustness of the entire network. Arweave is fully aware that it does not align with market preferences, taking a conservative approach, not embracing the miner community, and experiencing a complete stagnation in its ecosystem. It upgrades the mainnet at minimal costs while continually lowering hardware thresholds without compromising network security.

Review of the upgrade path from 1.5 to 2.9

The Arweave 1.5 version exposed a vulnerability where miners could rely on GPU stacking instead of real storage to optimize block production chances. To curb this trend, version 1.7 introduced the RandomX algorithm, limiting the use of specialized computing power and instead requiring general CPUs to participate in mining, thereby weakening computational centralization.

In version 2.0, Arweave adopts SPoA, converting data proof into a concise path of Merkle tree structure, and introduces format 2 transactions to reduce synchronization burden. This architecture alleviates network bandwidth pressure, significantly enhancing node collaboration capabilities. However, some miners can still evade real data holding responsibilities through centralized high-speed storage pool strategies.

To correct this bias, version 2.4 introduced the SPoRA mechanism, which incorporates global indexing and slow hash random access, requiring miners to genuinely hold data blocks to participate in effective block production, thus weakening the effects of hash power stacking from a mechanistic standpoint. As a result, miners began to focus on storage access speeds, driving the application of SSDs and high-speed read-write devices. Version 2.6 introduced hash chain control over block production rhythm, balancing the marginal benefits of high-performance devices and providing a fair participation space for small and medium-sized miners.

Subsequent versions further enhance network collaboration capabilities and storage diversity: 2.7 adds collaborative mining and pool mechanisms, improving the competitiveness of small miners; 2.8 introduces a composite packaging mechanism, allowing large capacity low-speed devices to participate flexibly; 2.9 introduces a new packaging process in replica_2_9 format, significantly improving efficiency and reducing computational dependency, completing the closed loop of data-driven mining models.

Overall, Arweave's upgrade path clearly presents its storage-oriented long-term strategy: while continuously resisting the trend of power centralization, it aims to continuously lower the entry barriers to ensure the long-term viability of the protocol.

Walrus: Embracing Hot Data - Hype or Hidden Depth?

Walrus, in terms of design philosophy, is completely different from Filecoin and Arweave. Filecoin's starting point is to create a decentralized and verifiable storage system, at the cost of cold data storage; Arweave's starting point is to build an on-chain library of Alexandria that can permanently store data, at the cost of too few scenarios; Walrus's starting point is to optimize storage costs for hot data storage protocols.

Magic Modified Error Correction Code: Cost Innovation or Old Wine in a New Bottle?

In terms of storage cost design, Walrus believes that the storage overhead of Filecoin and Arweave is unreasonable, as both employ a fully replicated architecture. Their main advantage lies in the fact that each node holds a complete copy, providing strong fault tolerance and independence among nodes. This type of architecture ensures that even if some nodes go offline, the network still maintains data availability. However, this also means that the system requires multiple copies for redundancy to maintain robustness, thus driving up storage costs. Especially in Arweave's design, the consensus mechanism itself encourages nodes to store redundant data to enhance data security. In contrast, Filecoin offers more flexibility in cost control, but at the expense of potentially higher data loss risks in some low-cost storage options. Walrus attempts to find a balance between the two, aiming to control replication costs while enhancing availability through structured redundancy, thereby establishing a new compromise path between data availability and cost efficiency.

The Redstuff created by Walrus is a key technology for reducing node redundancy, derived from Reed-Solomon ( RS ) coding. RS coding is a very traditional erasure code algorithm, which allows the dataset to be doubled by adding redundant fragments ( erasure code ) for the purpose of reconstructing the original data. From CD-ROMs to satellite communications to QR codes, it is frequently used in everyday life.

Erasure codes allow users to take a block, such as 1MB in size, and "expand" it to 2MB, where the additional 1MB is special data known as erasure codes. If any byte in the block is lost, users can easily recover those bytes through the codes. Even if up to 1MB of the block is lost, you can recover the entire block. The same technology enables computers to read all the data on a CD-ROM even if it has been damaged.

Currently, the most commonly used is RS coding. The implementation method is to start with k information blocks, construct the associated polynomial, and evaluate it at different x coordinates to obtain the encoded blocks. Using RS erasure codes, the probability of randomly sampling to lose large chunks of data is very low.

For example: A file is divided into 6 data blocks and 4 parity blocks, totaling 10 parts. As long as any 6 of them are kept, the original data can be completely restored.

Advantages: Strong fault tolerance, widely used in CD/DVD, fault-tolerant hard disk arrays ( RAID ), and cloud storage systems ( such as Azure Storage, Facebook F4).

Disadvantages: Decoding calculations are complex and have high overhead; not suitable for frequently changing data scenarios. Therefore, it is usually used for data recovery and scheduling in off-chain centralized environments.

Under the Decentralization architecture, Storj and Sia have adjusted traditional RS coding to meet the practical needs of distributed networks. Walrus has also proposed its own variant based on this - the RedStuff coding algorithm, to achieve lower cost and more flexible redundancy storage mechanisms.

What is the biggest feature of Redstuff? By improving the erasure coding algorithm, Walrus can quickly and robustly encode unstructured data blocks into smaller shards, which are distributed and stored in a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data block can be quickly reconstructed using partial shards. This becomes possible while maintaining a replication factor of only 4 to 5 times.

Therefore, it is reasonable to define Walrus as a lightweight redundancy and recovery protocol redesigned around a Decentralization scenario. Compared to traditional erasure codes ( such as Reed-Solomon ), RedStuff no longer pursues strict mathematical consistency but instead makes realistic trade-offs regarding data distribution, storage verification, and computational costs. This model abandons centralized scheduling.