High-growth start-up in digital assets market provides real-time pricing and market insights with Aiven for Apache Kafka®
Aiven makes setting up cloud databases so simple anyone can do it. Our set-it-and-forget-it solutions take the pain out of cloud data infrastructure.
As a high-growth start-up in the rapidly evolving marketplace of digital assets, Digital Asset Research (DAR) is setting new standards for digital asset data transparency and quality. To be competitive, DAR must handle huge volumes of data at speed to provide its digital asset prices and research in real-time, 24/7. It cannot afford downtime. However, the company was experiencing outages with its former Apache Kafka event streaming platform so it moved to Aiven for Apache Kafka®. DAR now has a highly resilient, scalable streaming data infrastructure helping to ensure the smooth delivery of its services as it continues to grow.
DAR is a specialized provider of ‘clean’ digital asset data, insights, and research for institutional clients such as Bloomberg, FTSE Russell, Wilshire and Chainlink. Each day, DAR collects data from 450+ exchanges and processes 200+ million trades to calculate institutional-quality prices on 10,000+ digital assets. It has a robust methodology for removing non-economic, noisy data that interferes with the calculation of true benchmark prices for digital assets. Established in 2017, this high-growth start-up has ambitious plans to expand its services, customer base, and international presence.
The digital asset market is volatile and changes quickly, so staying up to the second is a must for institutions. DAR must therefore provide asset pricing and solutions in real-time, 24/7 to work with top-tier institutional clients and to give investors the data they need to make decisions. DAR refreshes ticking prices at any interval including at granularity as low as 400 milliseconds.
The continuous delivery of these services requires highly resilient IT capable of handling the high-speed throughput of huge volumes of data and queries. However, 18 months ago DAR was facing issues in its database infrastructure which posed a serious risk to the uninterrupted delivery of its services.
At the time, DAR was using a MySQL-compatible, relational database engine in AWS. The solution was unable to handle the 20 million trades running each day and was frequently experiencing outages. So DAR migrated to SingleStoreDB Cloud and then later to a managed Apache Kafka® service from a different provider, both on AWS.
While SingleStoreDB brought many improvements to the database, issues arose with the Kafka solution. The clusters stopped working on multiple occasions and little technical support was offered from the Kafka provider.
“Every single time there was an upgrade to our Apache Kafka platform, it would go down,” says Soi Meng Tai, Director of Infrastructure, DAR. “There was a lack of root cause analytics for incidents. One time it was down repeatedly in the same day so we provisioned a new cluster, but that one experienced the same issues within an hour or two. During another outage, we waited eight hours for a response from the provider. It was a nightmare.”
If DAR did not address the outages, it would potentially miss client SLAs, incur costs and face serious reputational damage. “We calculate the pricing from all the available data on a one second interval, aggregate it, do the analysis and send it to the client,” says Michael Schmidt, Director of Engineering, DAR. “If the system goes down for just one minute, the clients don’t receive updated pricing. We lose 60 data sets. That historical data needs to be recovered as it informs our pricing algorithms. We’d need to recover it from the Exchanges which is really time-consuming and expensive.”
DAR also struggled with the unpredictability of data egress costs associated with the Kafka solution which were billed in arrears. “On a high trading day, we’d get destroyed with egress charges we’d been running up without knowing. It was impossible to manage,” says Schmidt.
“It was so easy, we decided to move our entire Kafka infrastructure to Aiven. Within three hours we were completely in production — without talking to anybody at Aiven. It was astonishing!”
Michael Zimberg
Chief Technology Officer
With SingleStoreDB working well in AWS, and with a lot of data in Amazon Simple Storage Service (Amazon S3), DAR began exploring alternative data streaming services hosted on AWS. DAR discovered AWS partner Aiven, and started experimenting with Aiven for Apache Kafka on AWS.
The DAR team was able to set up an account, spin-up a service and migrate a topic in minutes—all in self-service mode. “It was so easy, we decided to move our entire Kafka infrastructure to Aiven,” says Michael Zimberg, Chief Technology Officer, DAR.
At the time, DAR had more than 100 applications using 25 Kafka topics and was sending approximately 200 million messages a day to Kafka. Zimberg continues, “Within three hours we were completely in production—without talking to anybody at Aiven. It was astonishing!”
Since migrating to Aiven for Apache Kafka, DAR has seen very high levels of uptime and resiliency. Other than some minor issues at the outset, which were quickly diagnosed as sizing errors by Aiven tech support and fixed in a couple of clicks by DAR, the service is delivering 99.99% uptime. “Aiven for Apache Kafka provides incredible resiliency which, given the importance of uptime to our business, is invaluable to us,” says Zimberg.
The new service costs 25% less than the previous Kafka platform but, more importantly to DAR, the costs are predictable. Aiven provides all-inclusive pricing which removes the worry about excess costs for egress. Additionally, because it’s a managed service, DAR avoids the cost of hiring in specialist skills.
“We could have installed Kafka on bare metal EC2s and run it ourselves which would have been cheaper,” says Zimberg. “For that, I’d need a round-the-clock infrastructure team. With Aiven, we’ve avoided the need to hire maybe three full-time Kafka experts. That’s a massive cost saving and a headache that goes away for me.”
DAR has been particularly impressed with the quality of tech support: “Aiven consistently gives great advice and the team is always available to resolve any issues,” says Tai. “That’s why we have invested in the Priority Support service.”
By shifting from the previous MySQL-based solution to event streaming with Aiven for Apache Kafka paired with SingleStoreDB, DAR has seen a great improvement in performance. 18 months ago, MySQL couldn’t cope with DAR’s 20 million trades/day. Today, the Aiven service supports 200 million trades/day, and recently comfortably handled peak trading of one billion/day.
When DAR relied on MySQL, the company’s services were at best about one minute behind real-time, and at worst a few minutes behind. Now, despite the huge volume increase for ingress, DAR’s services operate tens of milliseconds behind real-time.
“We’re in an amazing position, and we’re growing fast. A lot of that is because we are very resilient — we don’t have outages that impact our clients. With Aiven we have an incredibly fast, robust streaming data infrastructure which helps ensure our services are available around the clock,” says Zimberg.
As DAR grows and expands its product offerings, the amount of data will inevitably skyrocket. In parallel, DAR’s use of Kafka will increase and Aiven will enable DAR to easily scale topics, compute and memory as required.
Zimberg expects DAR to be cross-cloud in the future, running workloads across the key hyperscalers. “It’s great that Aiven is compatible with all the large cloud service providers, so that removes a future planning headache. It will remain as a key part of our infrastructure, helping us to keep delivering on our ambitious business growth plans.”
Fireside chat
In this fireside chat, Mike Zimberg, CTO at Digital Asset Research, shares how they achieved a 99.99% uptime, while lowering their managed Apache Kafka® costs by 25% and improving their performance by 10x.
