Building an infrastructure that supports changing business needs
Hookdeck has grown quickly since launching in 2021, and its technology infrastructure has had to adapt and scale accordingly. A Google Cloud Platform user from the outset, Hookdeck initially used Cloud SQL for PostgreSQL® as its fully managed database service. However, it was soon in need ofPGBouncer,a lightweight connection pooler for PostgreSQL, which at the time was not offered by Google. Hookdeck discovered the Aiven for PostgreSQL® service which offers connection pooling utilizing PGBouncer and signed up.
“I’m passionate about open source and I immediately liked Aiven because it takes open source and productizes it with minimal changes and without creating any lock-ins,” says Maurice Kherlakian, Founding Engineer and CTO, Hookdeck.
The company was also attracted by the Aiven Cluster startup program which provided technical expertise and a credit of US $100,000 when Hookdeck joined. “The Aiven Cluster startup program was a big help for us. Tapping into Aiven’s expertise and having monetary credits offers tremendous value to young companies getting off the ground,” says Kherlakian.
Hookdeck used Aiven for PostgreSQL on Google Cloud as its primary data store, tracking connections, user profiles, webhook state transitions and more. It also used PostgreSQL to display analytics data to customers through dashboards.
However, as Hookdeck grew, the limitations of using PostgreSQL to store historical data and as an analytical database became more pronounced. “The tables it was handling had up to 500 million rows, and that was just with the last 30 days of data,” says Kherlakian. “PostgreSQL doesn’t operate well with datasets that large and the storage cost was increasing.”
Kherlakian’s team opted to move the historic data out of PostgreSQL and into Google BigQuery, a cloud data warehouse. BigQuery had the capacity to store data from the first day of Hookdeck’s history and to run business intelligence queries on the occasions that Hookdeck required. The company needed a highly available data streaming solution to transfer the data from PostgreSQL to BigQuery. It was keen on Apache Kafka and, already convinced of the value of using Aiven’s managed services, it chose Aiven for Apache Kafka®.
Part of the appeal of Kafka was the rich ecosystem of Kafka connectors, and Aiven helped Hookdeck take advantage of this. “By using Aiven for Apache Kafka® Connect, we could tap into the great tooling that’s available. We spent some time fine-tuning the integrations and ended up with staggering performance levels. We were able to do 60,000 messages a second,” says Kherlakian.