Aiven for Clickhouse® delivers 30x performance uplift and halves storage requirement
The team also started using fully managed Grafana® and OpenSearch® from Aiven to gain metrics and insights into the performance of its portfolio of Aiven services.
Aiven makes setting up cloud databases so simple anyone can do it. Our set-it-and-forget-it solutions take the pain out of cloud data infrastructure.
Hookdeck is a fast-growing tech start-up with ambitious goals. It provides a webhook infrastructure that handles the ingestion and error handling, ensuring users receive all their webhooks. The company has secured US $2.4 million in seed funding and has a worldwide customer base already. As Hookdeck has grown, the amount of webhooks and associated data it must handle has exploded too. To handle this, it has built a fast, reliable and scalable cloud data infrastructure using a broad portfolio of managed services from Aiven. This includes Aiven for Clickhouse®, a cloud data warehouse which analyzes huge amounts of data in real time for Hookdeck’s customers. This solution has delivered a 30x performance uplift while halving the associated storage requirement and bringing clear cost saving benefits.
Launched in 2021, Hookdeck is a fast-growing Québec-based start-up poised to disrupt the tech world. The company’s goal is to build the default solution for sending and receiving asynchronous event data between systems. In 2022 it raised US $2.4 million in seed funding and it already boasts hundreds of customers worldwide including Canal, Contra, Gorgias and Gemini.
Hookdeck solves a webhook challenge facing many developers. Webhooks are automated messages sent between applications, facilitating the sharing of information in real time amongst online platforms. Usually, a consumer will register with a provider, and when an event occurs on the provider side, they will issue a webhook containing some information about that event. This avoids the need for consumers to poll the provider's API, putting unnecessary strain on their servers. Historically developers have struggled with the lack of visibility into webhook failures and the subsequent limited ability to fix them, as well as with handling traffic spikes. Hookdeck fixes this by providing a webhook infrastructure that handles the ingestion and error handling, ensuring users receive all their webhooks.
Hookdeck has grown quickly since launching in 2021, and its technology infrastructure has had to adapt and scale accordingly. A Google Cloud Platform user from the outset, Hookdeck initially used Cloud SQL for PostgreSQL® as its fully managed database service. However, it was soon in need ofPGBouncer,a lightweight connection pooler for PostgreSQL, which at the time was not offered by Google. Hookdeck discovered the Aiven for PostgreSQL® service which offers connection pooling utilizing PGBouncer and signed up.
“I’m passionate about open source and I immediately liked Aiven because it takes open source and productizes it with minimal changes and without creating any lock-ins,” says Maurice Kherlakian, Founding Engineer and CTO, Hookdeck.
The company was also attracted by the Aiven Cluster startup program which provided technical expertise and a credit of US $100,000 when Hookdeck joined. “The Aiven Cluster startup program was a big help for us. Tapping into Aiven’s expertise and having monetary credits offers tremendous value to young companies getting off the ground,” says Kherlakian.
Hookdeck used Aiven for PostgreSQL on Google Cloud as its primary data store, tracking connections, user profiles, webhook state transitions and more. It also used PostgreSQL to display analytics data to customers through dashboards.
However, as Hookdeck grew, the limitations of using PostgreSQL to store historical data and as an analytical database became more pronounced. “The tables it was handling had up to 500 million rows, and that was just with the last 30 days of data,” says Kherlakian. “PostgreSQL doesn’t operate well with datasets that large and the storage cost was increasing.”
Kherlakian’s team opted to move the historic data out of PostgreSQL and into Google BigQuery, a cloud data warehouse. BigQuery had the capacity to store data from the first day of Hookdeck’s history and to run business intelligence queries on the occasions that Hookdeck required. The company needed a highly available data streaming solution to transfer the data from PostgreSQL to BigQuery. It was keen on Apache Kafka and, already convinced of the value of using Aiven’s managed services, it chose Aiven for Apache Kafka®.
Part of the appeal of Kafka was the rich ecosystem of Kafka connectors, and Aiven helped Hookdeck take advantage of this. “By using Aiven for Apache Kafka® Connect, we could tap into the great tooling that’s available. We spent some time fine-tuning the integrations and ended up with staggering performance levels. We were able to do 60,000 messages a second,” says Kherlakian.
At this point, Kherlakian’s team still needed to move the data for analytical queries from PostgreSQL so it started evaluating BigQuery, Elasticsearch and Clickhouse®. BigQuery was quickly discounted because the compute costs incurred by analytics was based on customer usage, and was therefore difficult to predict.
The team then looked at Elasticsearch and Clickhouse, focusing on the query performance, support for full text search and integration with Kafka. Clickhouse emerged as the best technology. So Hookdeck chose Aiven for Clickhouse® as its analytical database, using it to power the customer dashboards displaying data about their webhooks, for example, a histogram showing how many webhooks have been received in a certain period.
Harnessing data with Aiven for Clickhouse®: Unveiling webhook activity through captivating histograms
With Aiven for Clickhouse in place, Hookdeck has a fast, open-source cloud data warehouse that allows it to analyze huge amounts of data in real time. “The performance of Aiven for Clickhouse is seriously impressive. Previously, the histograms for the larger workspaces would take around 30 seconds to load. Now it takes less than a second,” says Kherlakian.
Thanks to its excellent data compression ability, the storage requirements and costs have been significantly reduced. The data in Clickhouse uses 700 gigabytes of storage, compared to the 1.5 terabytes required in PostgreSQL.
By using a broad range of Aiven services, Hookdeck has built a reliable, integrated data infrastructure on Google Cloud which leverages the best of open source technologies. “There are huge benefits from getting multiple services from Aiven,” says Kherlakian. “We only have one vendor to onboard, everything is compliant and billing is consolidated. From an engineering perspective, we’re dealing with something that’s familiar and, because all the services are integrated, adding another one is really easy.”
The scalability of the cloud data infrastructure that Aiven provides is another important benefit as the company’s rapid growth trajectory continues. The number of webhooks Hookdeck processes is significantly increasing every month and as the corresponding influx of data flows through its infrastructure, Hookdeck can easily scale its Aiven services, increasing storage, nodes and clusters as required. “I can think of quite a few times when our database has been saturated and we needed a bigger instance. In the click of a button we had it,” says Kherlakian.
Of course, the infrastructure the engineering team builds must have high availability—downtime of any applications would be a disaster for Hookdeck and its growing customer base. With 99.99% uptime guaranteed by Aiven, Hookdeck’s customer SLAs as well as its reputation are protected.
Kherlakian’s engineering team is small, so efficiency is key. When evaluating a new technology, his team must consider what it will take to maintain it in the future. “Aiven not only makes it easy to implement, but its support team takes all the stress out of managing it,” says Kherlakian. “That allows our team to focus on building the best possible webhook infrastructure.”
The start-up currently has seven people but is actively recruiting to help scale its platform and support the anticipated growth in business. “We currently handle around half a billion webhooks a month and we’re well on our way to billions. My engineering team has a mission to create an architecture that supports tens of billion,” says Kherlakian. “As an organization, we’re driven to build services that enable developers across the world to focus more on creating what they love, while relying on us to ingest and process their event data. It’s great to have Aiven with us on our journey.”
