ActiveFS | Create efficient and scalable distributed storage with data redundancy, rebalancing, and resharding

Lightweight library for building efficient and scalable distributed file storage

What is ActiveFS?

ActiveFS is a Java library that allows to create distributed file storage supporting data redundancy, rebalancing and resharding. It provides an asynchronous abstraction on top of both local FS and remote FS with a simple FTP-like protocol with zero-overhead streaming.

All data in ActiveFS is organized in file systems. The key principle is effective data immutability, which enables simple design, low-overhead and aggressive caching. This approach allows to create high-throughput data storage and processing applications.

Create distributed file storage

ActiveFS is an efficient and lightweight alternative for kernel-space distributed FS like Ceph or Apache Hadoop. It allows to create combined, fault-tolerant cluster FS with repartitioning and rebalancing support. Moreover, ActiveFS contains a set of predefined communication strategies that can be simply combined.

private RemoteFsClusterClient client;

    ...

Eventloop eventloop = Eventloop.create();

Map<Object, FsClient> clients = ...;

client = RemoteFsClusterClient.create(eventloop, clients)
    .withReplicationCount(N / 2)
    .withServerSelector(RENDEZVOUS_HASH_SHARDER);

How does it work?

It is quite hard to implement distributed POSIX file systems. Existing solutions are either not fault-tolerant enough or have too much overhead. Moreover, they require coordinating centers which slow down and overhead the system.

To overcome these drawbacks, ActiveFS was inspired by P2P file systems like BitTorrent and IPFS. ActiveFS moves to modern and efficient append-only semantics without coordinating centers. With ActiveFS you have an application-embedded lightweight file storage library that is production-ready, supports cluster P2P or client-server communications and dynamic servers substitutions.

Add ActiveFS to your project

Add ActiveFS Maven repository to your project. Please note that ActiveFS depends on tiny active-jmxapi tool (optional) and ActiveJ platform.

<dependency>
    <groupId>io.activej</groupId>
    <artifactId>activej-fs</artifactId>
    <version>1.0</version>
</dependency>

Origins

ActiveFS is one of the ActiveJ technologies that also include lightning-fast serializers, bytecode generators, dependency injection and others. To learn more about ActiveJ ecosystem, visit website of the core ActiveJ technology - ActiveJ framework.