Navigating Kafka: Challenges, solutions, and the future of real-time data streaming

In this Data (R)evolution episode, Matan Mizrahi and Filip Yonov join us to discuss how Kafka effectively handles and optimizes data streams, the pursuit of standardized APIs, and the vital role of AI in optimizing complex systems. They also share how AI and open-source tools are transforming the landscape. Tune in to hear about the evolution of data streaming and the transformative impact of AI and cloud-native solutions.

Key takeaways:

  1. Emphasize the need for customer-defined scaling logic based on metrics and auto-scaling events for efficient Kafka management.
  2. The importance of integrating AI, like the Ivan AI database optimizer, to enhance workload optimization and complex system management in Kafka.
  3. The significance of strategic partner relationships in providing advanced software solutions and necessary knowledge and experience to mitigate risk in critical data infrastructure projects.

Podcast series: Data (R)Evolution

Topics around data, innovation and economics in a rapidly-changing cloud world.

Data drives innovation and revolutionizes the tech world. Data (R)evolution, powered by Aiven, explores the intersections of data, AI, innovation, and economics through the ever-evolving landscape of cloud technology. Led by hosts Francesco Tisiot and Janki Patel Westenberg, navigate this digital revolution in conversation with experts at the forefront of change.

Subscribe and never miss an episode

Watch the series

Hosts

Francesco Tisiot

Field CTO & Head of DevEx, Aiven

Francesco Tisiot, Field CTO & Head of DevEx at Aiven, is dedicated to helping customers achieve success through data-driven innovation. Francesco believes that technology should be an enabler, not a barrier, and works closely with clients to understand their unique needs and develop tailored solutions that deliver measurable results. A seasoned technologist with a proven track record, Francesco is a trusted advisor to organizations across industries.

Janki Patel Westenberg

Head of Product Growth and Marketing

Over the last 10 years, Janki has successfully led technical partnerships collaborating with Google Cloud and AWS to build database and caching solutions for gaming and media. Additionally, she has spearheaded GTM strategy partnered with AWS' product leadership for 19 AWS Analytics and Database services for multiple industries. Janki also is a proud alumna from University of Wisconsin-Madison (Go Badgers!)...fun fact, she studied Genetics and did research before moving to the cloud industry!

Guests

Maja Schermuly

Solution Architect

Maja is a Solution Architect at Aiven, specializing in designing and implementing cloud data architectures. Her hands-on experience allows her to enable customers to build scalable solutions that are both practical and effective.

Matan Mizrahi

Founding Engineer, Stealth Startup | Former Senior DevOps, Claroty

Matan is a founding engineer at a stealth-mode startup and one of Israel’s most recognized experts on Kafka and large-scale cloud infrastructure. At Claroty, he led Kafka optimization initiatives and implemented DevOps practices that drove major cloud cost savings. Known for his deep technical expertise and engaging stage presence, Matan brings battle-tested strategies for reducing TCO in real-time, data-intensive environments.