Aiven is connecting streaming data with the Google BigQuery ecosystem

Organizations today require real-time analysis to gather immediate feedback and improve their product or service offering. Research shows it takes only a mere three-second window to retain a customer on a website. Within this brief span, responding to data in real time becomes imperative for grasping customer insight and delivering customized offerings in a matter of seconds. To better understand the needs of the customer, data warehousing technologies are now leveraged in a very different capacity: one that allows developer teams to tap into the power of real time data and provide low latency analytics.

Stream processing technologies, like Apache Flink®, have emerged as a key enabler for real time data computations, allowing organizations to make better sense of their data and bring millisecond latency response to their users. An example is Pinterest, which runs experiment analytics in real time with the power of streaming technologies like Apache Kafka® and Apache Flink. Leveraging the combination of a robust streaming infrastructure with scalable data warehousing becomes a de facto combination for turning your customer response from hours - or even days - to just seconds.

Today we are thrilled to share two exciting announcements that tightly connect Aiven’s streaming technologies with Google's BigQuery ecosystem for data warehousing.

Firstly, we are excited to announce that Aiven has successfully achieved Google Cloud Ready - BigQuery designation which marks a significant milestone for our partnership with Google Cloud. This designation enables customers to have better confidence in the products that they use today being integrated optimally with BigQuery.

Additionally, we are introducing the early availability of the open source connector for Apache Flink® to BigQuery allowing more low latency analytical workloads on top of streaming data directly on Google Cloud!

Joining the Google Cloud Ready - BigQuery Initiative

We are pleased to announce that we have successfully achieved Google Cloud Ready - BigQuery designation. This designation enables customers to easily discover Aiven technologies directly in the BigQuery Partner Center.

By earning this designation, Aiven has proven that our products have met a core set of functionality and interoperability requirements when integrating with BigQuery. Being part of the program also ensures more opportunities for Aiven and Google Cloud’s partner engineering and BigQuery teams to collaborate closely.

“We are very excited that Aiven for Apache Kafka is now recognized as a partner of the Google Cloud Ready - BigQuery program, as this is a testament to our continuous collaboration and partnership with Google Cloud and its ecosystem of technologies to improve and unlock new use cases for our users,” said Jonah Kowall, Vice President of Product Management at Aiven.

“The Google Cloud Ready-BigQuery designation gives customers confidence that solutions have gone through a formal certification process and will deliver the best possible performance with BigQuery,” said Ritika Suri, Director of Technology Partnerships at Google Cloud. “With Aiven, customers can connect all of their data and metrics with BigQuery to more easily optimize their business performance.”

Swift is one of the customers implementing the company’s Warehouse Management System 2.0: using Aiven for Apache Kafka with Apache Kafka Connect to stream warehouse data into BigQuery. Using Aiven for Apache Kafka with BigQuery allows them to implement advanced horizontal logistics collaboration (HLC) with the larger ecosystem of their parent organization. "With the great help and support from Aiven, we can keep our infrastructure stable, scale our business fast, and keep control of our data," says Alfi Zainuddin, CTO at Swift Solutions

Hookdeck, our joint customer with Google Cloud, has successfully used Aiven for Apache Kafka and its highly available data streaming solution to transfer the data from PostgreSQL to BigQuery. “....we could tap into the great tooling that’s available. We spent some time fine-tuning the integrations and ended up with staggering performance levels. We were able to do 60,000 messages a second,” says Maurice Kherlakian, CTO & Founding engineer at Hookdeck.

Bridging the gap between streaming data and analytical workloads with the connector for Flink and BigQuery

In addition to the Google Cloud Ready - BigQuery designation we are also introducing the open source connector for Flink and BigQuery. Aiven is the trusted open source data platform for everyone! In line with our vision, we want the integration between our platform and popular technologies to be as easy and streamlined as possible giving additional flexibility not only to our customers but also to the wider open source community wanting to integrate Apache Flink with popular technologies like BigQuery.

In line with this, we are bringing the Flink to BigQuery sink connector licensed under the Apache-2.0 software license that allows developers to use the connector no matter whether they use open source Apache Flink or a fully-managed service such as Aiven for Apache Flink.

Users of the new Flink and BigQuery connector can now develop scalable streaming data pipelines and streaming applications by reading data from multiple sources including data streaming technologies like Apache Kafka® or by leveraging Change Data Capture (CDC) to record row-level changes from their operational databases like PostgreSQL, in real time. With a stream processing technology like Apache Flink, developer teams can easily make real time transformations to the data and pass the transformed events to BigQuery for further analytical and data warehousing applications and use cases.

The existing early availability of the connector currently supports Apache Flink’s SQL & Table API while future iterations will add support for Flink’s DataStream API and BigQuery’s new Storage API for more efficient and cost-effective analytical workloads at scale.

Pricing & availability

The sink connector between Apache Flink and BigQuery is available as an integration in the Aiven Console today. Simply go to your Aiven for Apache Flink application and select the BigQuery integration from the Aiven Console or through the Aiven CLI. Users of the connector can now seamlessly integrate their streaming data with the BigQuery ecosystem in Google Cloud regions around the globe. Explore more options on our pricing page.

The connector can also be accessed directly on GitHub. We encourage more contributions and ideas from the users of the connector and the wider Flink and BigQuery communities.

Get Started with Aiven for Apache Flink

You can now seamlessly integrate Aiven for Apache Flink with BigQuery with just a few clicks in the Aiven console or commands in Aiven’s CLI. If you are new to Aiven, you can sign up for a free 30-day trial or schedule a demo with our team and explore how Flink and BigQuery can work together for you.