Discover the power of Dragonfly: Scalability, resilience, and real-time messaging. Uncover the benefits of this cutting-edge in-memory data store solution now!
Joe Zhou
|RSS FeedJoe Zhou is a Developer Advocate at Dragonfly. He is a guest contributor on the Aiven blog.
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
Dragonfly is a performant and memory-efficient drop-in replacement for Redis®, utilizing a modern multi-threaded shared-nothing architecture. That’s it! In essence, we have already captured the ambitious mission of Dragonfly. Yet, to fully appreciate the significance, let’s unpack this description and examine the benefits and key features in detail.
Redis is one of the most popular in-memory data stores used in countless web server applications. In order to provide a seamless developer experience, Dragonfly is designed to be fully compatible with Redis in terms of the wire protocol. At the time of writing, Dragonfly supports 240+ of the most commonly used Redis commands. By using the same SDKs and tools you are already familiar with, switching to Dragonfly would not require changing existing code, except for the client connection credentials. As an added bonus, Dragonfly supports running in Memcached mode as well, which also provides full compatibility with the Memcached wire protocol and APIs.
Leveraging a multi-threaded, shared-nothing architecture, Dragonfly offers unparalleled performance enhancements over traditional in-memory data stores. Unlike Redis, which mostly employs a single-threaded design, Dragonfly’s architecture is optimized to fully utilize modern multi-core servers, delivering significantly higher throughput. Based on our recent benchmarks, Dragonfly is able to achieve 6.43 million operations per second when running on a single Amazon c7gn.16xlarge instance, equipped with the Graviton 3E processors. Below, we are showing the maximum operations (GET and SET commands) per second when running Dragonfly with different numbers of threads, restricting P99.9 latencies to being under 10 ms.
Max Opeations Per Second vs. Dragonfly Threads
While many of the existing Redis data structures are reused for their efficiency and robustness, Dragonfly continues the innovation and introduces advanced modern data structures that optimize memory usage, ensuring more efficient data storage and management. One example is the B+ tree based Sorted-Set implementation in Dragonfly, which observes up to 40% memory reduction for Sorted-Set data entries. Furthermore, Dragonfly also has a brand new algorithm for snapshotting, which ensures stable memory usage throughout the whole snapshotting process. These advancements represent Dragonfly's commitment to pushing the limits of being the most advanced in-memory data store.
Offering the same Redis or Memcached developer experience, Dragonfly aims to revolutionize the in-memory data store landscape with a superior data and operational management experience for much more demanding workloads.
Dragonfly works with both basic and complex Redis data types, including String, List, Set, Sorted Set, Hash, Pub/Sub Messaging, Stream, and Geospatial Index. Additionally, Dragonfly natively supports the popular JSON data type. It also offers powerful search features, enabling both faceted search and vector search.
As mentioned above, Dragonfly is able to fully utilize powerful multi-core servers. With easy vertical scalability, Dragonfly is comfortable handling up to 1TB of in-memory data. While we believe this should cover a vast majority of the workloads, multi-node cluster mode is also supported with dynamic horizontal scalability under active development.
Dragonfly Single Instance Can Handle Much Larger Workloads
Snapshotting in Dragonfly is based on versioning and asynchronous serialization. Unlike the conventional approach, which is based on copy-on-write memory management, Dragonfly doesn't spike memory usage during snapshotting on write-heavy instances. In the meantime, snapshots can be directly saved to cloud-native storage systems such as Amazon S3 or via Kubernetes persistent volumes.
To ensure reliability and seamless failover, high-availability through an eventually consistent primary-replica model is also supported. This can be achieved by using the REPLICAOF command, Redis Sentinel, or the native Dragonfly Kubernetes operator.
Similar to Redis, Dragonfly serves as an efficient and dependable centralized caching solution for server-side applications, enhancing performance by reducing the need to frequently access slower, disk-based databases.
Similar to Redis, Dragonfly serves as an efficient and dependable centralized caching solution for server-side applications, enhancing performance by reducing the need to frequently access slower, disk-based databases.
Although in-memory data stores typically struggle with complex analytical queries, Dragonfly is equipped with specific data types (such as Bitmap, HyperLogLog, and Set) designed for measuring unique counts and gathering statistical information. These data types are perfect for tracking unique page visits or managing real-time leaderboards.
As highlighted earlier, Dragonfly excels at providing faceted search and vector search right out of the box, making it ideal for creating advanced search functionalities within applications.
Dragonfly not only facilitates efficient real-time messaging, perfect for applications needing instant data updates or chat functionalities, but it's also adept at background job processing. We are actively integrating and optimizing frameworks in this area, including BullMQ and Sidekiq, to ensure seamless operation and high performance for tasks scheduled to run in the background.
As of March 20th, 2024, Aiven, the open-source cloud data platform company, announced the general availability of Aiven for Dragonfly®. This service brings all the powerful features of Dragonfly to your cloud, offering a scalable and fully managed solution.
Explore and sign up for Aiven for Dragonfly today, and take full advantage of the solid functionalities that meet your business needs.
Learn moreYes, Dragonfly is compatible with Redis® in terms of the wire protocol, both RESP2 and RESP3, as documented in the specification. Existing Redis clients, SDKs, and GUI desktop managers can directly connect to Dragonfly. As of April 2024, Dragonfly supports more than 240 Redis commands. In short, if Dragonfly supports the Redis commands you use, then your existing Redis code should work with Dragonfly out of the box.
Dragonfly is a community-driven, source-available project hosted on GitHub. Dragonfly source code is available under the Dragonfly Business Source License (BSL). In short, developers are free to use Dragonfly for any use case other than offering a hosted in-memory data store. The license states: You may make use of Dragonfly: (i) as part of your own product or service, provided it is not an in-memory data store product or service; and (ii) provided that you do not use, provide, distribute, or make available the licensed work as a service.
Aiven for Dragonfly is a hosted in-memory data store service, which means that Dragonfly is licensed to Aiven under a different license based on collaboration and mutual agreement.
An in-memory database (or data store) is a type of database that primarily stores its data directly in the main memory (RAM) of a computer rather than on disk-based storage. This approach allows for faster data retrieval and processing compared to disk-based databases because accessing data in RAM is significantly faster than reading data from a disk. In-memory databases are normally used as a caching layer in a backend system, but they can handle other use cases as well. They are well-suited to applications where performance and low latency are critical. Some in-memory databases also provide on-disk storage options for backup and fast recovery purposes. Memcached, Redis, and Dragonfly are examples of in-memory databases.
A NoSQL database is a type of database that provides a mechanism for storage and retrieval of data that is modeled in a way other than the tabular relations used in relational databases. Unlike traditional relational databases that use tables, rows, and columns to store data, NoSQL databases can store structured, semi-structured, or unstructured data, making them more flexible in terms of the types of data they can handle. As suggested by the name, NoSQL databases often do not expose SQL as the querying and data manipulation interface to developers. We can further categorize NoSQL databases into key-value stores, document databases, wide-column databases, graph databases, and other specialized data stores. Under this categorization, Dragonfly can be primarily considered a key-value and document store, with additional real-time, statistics, and search capabilities.
Max Opeations Per Second vs. Dragonfly Threads
Dragonfly Single Instance Can Handle Much Larger Workloads