Brolly uses Aiven for Apache Kafka^® to make sense of its customer’s social data

As a social media archiving & insights platform, effective data processing and data management are mission-critical for Brolly. Aligned with rapid customer growth in social media traffic, Brolly needed a robust way to collect and store these social media conversations – one that was compliant with industry data privacy and security regulations. Brolly partnered with Aiven to create a modern data infrastructure that is resilient, secure, and agile enough to keep ahead of customers’ expanding social media archiving needs. Using Aiven for Apache Kafka®, Brolly has significantly contributed to reducing system errors and operating costs, while also freeing up valuable IT resources. This enables its people to focus on creating new services that matter most – value creation and customer satisfaction.

Growing on social media, responsibly

According to Statista, in 2022, consumers spent an average of 147 minutes per day on social media. Given the popularity of social media apps as a preferred communication tool, many large organisations like banks, insurers, and Government agencies regularly use social media to interact with their customers, every day. This near real-time communication generates vast amounts of data that needs to be processed and stored in line with industry compliance requirements.

Given that Brolly’s customers are mainly in regulated industries such as Government, financial services, education, and healthcare providers, collecting and keeping social media records secure is of utmost importance to them.

“Data is a core business function for us. Capturing and preserving social media conversations securely, reliably, and in compliance with the various data and privacy regulations across different regions within Australia and the region is critical for our customer base. Our analytics framework needed to be resilient enough to make sense of the massive amount of data that is generated,” says Ali Nazemian, Chief Technology Officer at Brolly.

“Data is a core business function for us. Capturing and preserving social media conversations securely, reliably, and in compliance with the various data and privacy regulations across different regions within Australia and the region is critical for our customer base. Our analytics framework needed to be resilient enough to make sense of the massive amount of data that is generated,” says Ali Nazemian, Chief Technology Officer at Brolly

Staying ahead to manage increasing data volumes in real-time

Prior to partnering with Aiven, Brolly was struggling to manage the endless streams of structured and unstructured information and data coming in from multiple sources. All too often, this was happening in real time, using an outdated data streaming architecture that was no longer fit for purpose. As a result, Brolly was struggling to manage the increased volume of real-time data and their transactional use cases, which resulted in additional time spent on computation. To make matters worse, the previous platform failed to capture certain customer data meaning valuable IT resources were spent recapturing the data. Due to the platform not being fully automated, this led to human intervention which went against regulatory data compliance.

The last straw was when Brolly’s engineering team found themselves drowning in a backlog due to the platform’s inability to ingest massive amounts of data, which frustrated the data users, engineers, and ultimately, customers. Support tickets were also slow to generate.

Brolly realised it needed to invest in a modern database and system that was agile and flexible enough to keep up with the large volumes of customer data needing processing and archiving. Its new platform needed to be robust enough to accurately capture, process, and archive its customers’ growing volumes of social media data. It also needed to be simple to use and easy to manage with the ability to ingest various incoming data streams and support multiple data sources.

“It was crucial that the team was familiar with the technologies powering the new data platform with zero vendor lock-in and managed by a service provider that we could trust to take care of the back-end services, while we focused on delivering value to our customers,” Ali says.

Data infrastructure made simple

To meet Brolly’s requirements for a modern data platform that could keep pace with its exponential growth in customer data, Brolly partnered with Aiven, selecting Aiven for Apache Kafka.

“We were looking for a reliable technology partner that we can trust to deliver, scale, and support our social archiving needs and services on demand, without additional resourcing pressures. Aiven made everything so simple for us from the get-go,” said Ali.

Aiven was able to provide all the data infrastructure components Brolly needed, coupled with the innate ability to scale horizontally and vertically to meet the company’s streaming needs and workloads, as it continues to accommodate greater amounts of traffic and use cases.

The digital transformation, including migrating Brolly customers’ data over to the new Aiven platform and services was completed in less than four months without any downtime, allowing Brolly to continue meeting their customers’ social media archiving requirements.

Currently, Brolly is running Apache Kafka as the backbone of its data infrastructure. Avro with Karapace is also used to circumvent the decoupled nature of Apache Kafka, ensuring seamless message compatibility amongst Brolly microservices. While Kafka Connect is used to share events internally amongst Brolly’s critical services.

Enjoying the best of both worlds with Aiven

Aiven’s cloud-based infrastructure is well-suited to support the growing scale of Brolly’s intensive social media archiving needs, such as the rate of data ingestion changing frequently, where a solution that can keep up with millions of messages over a short time is needed.
“Messages can go viral, so it’s important to have a scalable solution that can keep up with social media records no matter what the rate of ingestion is,” Ali elaborates. “To overcome this, we have used Aiven for Apache Kafka and its ecosystem to build an event-driven data pipeline that can be a highly distributed and scalable medium for social media records in transit.”

The event streaming platform’s flexible architecture also makes moving data a breeze, tapping its open-source advantages to pull in various APIs, management interfaces, and applications to give diverse options. As an open-source cloud data platform, Aiven connects all the components in the value chain, from producers to consumers.

“With the use of Aiven’s fully managed service for Apache Kafka, we have the best of both worlds by keeping our operations cost low, having the assurance of commercial support, and using a trusted and reliable solution with the support of the wider open-source communities. This platform support is vital for a dynamic outfit like Brolly,” Ali says, “as it allows Brolly to focus on what is important to us, which is giving our customers peace of mind.”

“With the use of Aiven’s fully managed service for Apache Kafka, we have the best of both worlds by keeping our operations cost low, having the assurance of commercial support, and using a trusted and reliable solution with the support of the wider open-source communities. This platform support is vital for a dynamic outfit like Brolly,”

Taking operational efficiencies and customer experience to the next level

Since implementing Apache Kafka, Brolly has been able to manage its resources better which sees a reduction in infrastructure total cost of ownership by 60%, with accuracy and data recovery improving by more than 300%.

While accuracy is paramount, the twin competencies of speed and reliable data volume handling are equally important. “When a social media account gets connected to Brolly, all the corresponding social media records for that account get collected in just a few hours in addition to serving real-time messages immediately,” the CTO said. “Our customers are now able to extract and search millions of records in real-time once their historical data is collected.”

As a result, Brolly’s customers can now increase their visibility, engagements, and customer traffic while adhering to the industry’s regulatory and compliance needs.

On the technical side, the record processing latency has decreased by two orders of magnitude from more than a minute to less than a second on average. There has also been a significant decrease on the computational side of Brolly’s infrastructure as they are now able to work more efficiently, increasing operational efficiencies by more than 200%.

The blueprint for social media archiving

Brolly is currently identifying new services for its customers to detect inappropriate content or suspicious events in real time. The company is also intending to up the ante to protect its customers’ data and their online communities by helping them reduce risks associated with potential social content violations while maintaining brand consistency.

“The use of open-source technologies from Aiven has helped us to focus on making social media a better place for our customers, communities, and citizens,” summarises Brolly’s Ali. “This will allow us to be the best social media archiving and insights platform and yet give back to the community.”

To get the latest news about Aiven and our services, plus a bit of extra around all things open source, subscribe to our monthly newsletter! Daily news about Aiven is available on our LinkedIn and Twitter feeds.

If you just want to find out about our service updates, follow our changelog.