Quickwit

Quickwit is an open-source search engine and server designed specifically for observability use cases, focusing on the efficient indexing and retrieval of logs and traces. The software operates as a server that can be deployed on Kubernetes via a helm chart or as a standalone instance to manage large volumes of telemetry data.

The engine is built from the ground up to search directly on cloud storage, including Amazon S3, Azure Blob Storage, and Google Cloud Storage. It utilizes a decoupled architecture where compute and storage are separate, allowing indexers and searchers to remain stateless. This design enables sub-second search speeds on cloud storage while reducing infrastructure costs by avoiding the need for expensive local disk arrays across all nodes.

Key features

• Compatible API for use with existing clients
• Full-text search and aggregation queries
• OTEL-native support for logs and traces
• Schemaless or strict schema indexing
• Native integration with Kafka, Kinesis, and Pulsar
• Multi-tenancy with partitioning and multiple indexes
• Retention policies and delete tasks for GDPR compliance
• RESTful API for system interaction

Quickwit is written in Rust and leverages the Tantivy library for its core indexing capabilities. It supports a large subset of common query DSL and ingest API, which allows users to migrate common log shippers such as Vector, Fluent Bit, or Syslog without significant reconfiguration. To ensure compatibility with strict clients, the node configuration includes an option to impersonate specific versions of other search engines through custom headers. The system is designed for high availability in search, while high availability for indexing is supported specifically when using a Kafka source.

It is a cost-effective solution for big data and log management that prioritizes cloud-native storage and stateless compute for scalability.