AutoMQ

AutoMQ is an open-source, self-hosted server that implements a shared storage architecture. It replaces traditional local disk storage with cloud object storage, such as Amazon S3 or MinIO, to create a stateless broker environment. This design allows for scaling the computing layer without the need for manual data movement or partition reassignment, which are typically complex processes in standard clusters.

The software is deployed as a server and is fully compatible with the relevant APIs, allowing existing producers and consumers to work without modification. It is designed specifically for cloud-native environments where reducing storage costs and eliminating cross-availability zone traffic fees are primary objectives. Users can deploy the system using Docker and Docker Compose for evaluation, development, or testing cluster features.

Key features

• S3-compatible storage integration for diskless operation
• Automatic scaling of stateless brokers in seconds
• Built-in auto-balancer for partition and traffic scheduling
• Rack-aware routing to eliminate cross-AZ traffic costs
• Native export of Prometheus and OpenTelemetry metrics
• Table Topic support for Apache Iceberg and S3 tables
• Full compatibility with standard streaming APIs
• Write-ahead log (WAL) to optimize S3 API costs

AutoMQ transforms the shared-nothing architecture into a shared storage model. It utilizes an S3 Storage Adapter to reimplement UnifiedLog, LocalLog, and LogSegment classes to create logs on S3 instead of local disks. To maintain performance, it employs a shared streaming library called S3Stream, which manages the WAL and utilizes LogCache and BlockCache to boost read efficiency. This architecture is intended for organizations running high-throughput streaming workloads in the cloud who want to avoid the operational complexity and high EBS storage costs associated with managing stateful brokers.

It is a cost-optimized, serverless streaming platform for cloud-first infrastructure.