Load testing your Apache Kafka cluster and consumer applications

Socrates preached, “To know thyself is the beginning of wisdom.” The same maxim applies to your Apache Kafka project. Hence, load testing is an important practice for optimizing your real-time data stream; engineers must understand the capacity and limitations of their project before deployment, failure to do so could be catastrophic (e.g. Robinhood’s outage on a historic trading day).

Load testing has three main objectives:

• Determine the system’s maximum capacity
• Determine the breaking points in your system
• Observe how the system will recover without manual intervention

You can test these measures through several types of load testing; a stress puts the system under a heavy load to observe its reaction; a spike or a surge tests the system’s reaction to a sudden spike in traffic, which is relevant for any business that faces seasonal traffic differences; finally, a soak checks the endurance of your stream.

Watch a recording where, David Esposito, an Aiven Solutions Architect, explores load testing in Kafka Office Hours, a recurring forum for Apache Kafka thoughtshare hosted by our partner Lenses.io.