The challenge
Founded in 2013, JobCloud provides traditional job-posting and job-searching services as well as multilingual semantic recruitment technologies, candidate management, employer branding tools, analytics and programmatic job ads.
For historical reasons, JobCloud had ended up running a number of legacy systems. “We have a lot of legacy systems in place, using outdated languages and implemented with a really strange database setup, and nobody dared to touch them,” says Nicolas Chiu, Lead Software Developer at JobCloud. The systems were just there, mostly working, but this was hardly ideal from a maintenance viewpoint.
JobCloud’s goal was to modernise their systems into an event-driven architecture to better support modern data requirements. At the same time, they really didn’t want to directly access the strange APIs. The easiest answer, it seemed, would be to insert a messaging layer between the other systems that would separate the complexity of the legacy systems from other services, whether existing or new.
After months of evaluating messaging and streaming solutions, JobCloud decided to go with Apache Kafka, arguably the quintessential solution for streaming event data. “It has a huge ecosystem and contributor base. It also has a lot of connectors, which makes it easy to integrate other systems,” Nick says.
But Apache Kafka is not the easiest system to run.
“When we started out, we noticed that Apache Kafka has a huge learning curve. There's just so many things that you can configure. And at the very beginning, we were really overwhelmed.”
Finding a provider wasn’t easy, either. The first one they tried couldn’t offer all the features they needed, like ACLs on topics and locking integration, in spite of the prohibitive price tag. The JobCloud systems also proved to be too much for the provider’s technical capabilities.
“We were desperate to find anything better, and back then it was really hard to find good providers,” Nick reminisces. “So when we found Aiven, we were immediately like 'hel-lo, what kind of plans do you have to offer?!'”