Edtech unicorn builds “do-it-all" system on Apache Kafka® and unlocks new phase of growth
Aiven makes setting up cloud databases so simple anyone can do it. Our set-it-and-forget-it solutions take the pain out of cloud data infrastructure.
In 2019, Simplilearn and its students were ready for massive growth — but its IT systems weren’t. Discover how the company unlocked its potential with Aiven for Apache Kafka®, solved its scalability headaches and helped millions of learners reach their career goals.
Simplilearn is the world’s No. 1 online bootcamp for digital skills. Its mission is to transform lives through education, and the company has helped over 5 million learners reach their career and business goals.
Based in Bangalore and San Francisco, Simplilearn has 2,000 qualified trainers running 3,000 classes each month for 75,000 learners on average. Between 2018 and 2021, it doubled its learner base, partly due to social changes driven by the pandemic.
“Our ambition is to develop our offering even further,” says Mrinal Barua, Senior Associate Director at Simplilearn. “We’re working with top-tier universities, corporations and industry bodies to provide new degree-level and postgraduate courses, so learners all over the world can get skilled, well-paying jobs in the digital economy.”
In 2019, Simplilearn was growing fast, with VC investment, partnerships with blue chip brands and rankings that put it ahead of some of the world’s leading universities for online provision.
But this expansion in the number of students and courses exposed some underlying issues in the company’s learning management system (LMS). All students at Simplilearn use this core platform to take their courses. As learners worked through their modules, their individual progress was synced to various databases using APIs. Only when all data was logged to the databases could users see their progress reflected on screen. And because there were so many connected systems, a full sync could take up to 30 minutes. This problem became especially noticeable when users had completed a course and went to get the certificate. The long waiting time for completion to register in the LMS led to confusion, user dissatisfaction and poor feedback.
As Simplilearn developed its offering towards more complex study programs, the problem became even more apparent. “When we started registering learners for multiple classes at once, this put further load on our systems,” says Kusum Saini, Senior Architect at Simplilearn. “The more we grew as a company, the more potential problems we saw with the user experience.”
Latency was also making it hard for Simplilearn to make quick business decisions. “Our marketing depends on real-time data from the LMS,” says Ritesh Wanchoo, Cloud Architect at Simplilearn. “We send users emails or nudges within the platform about other courses they can take to improve their skills. But without instant visibility into learners’ progress, we couldn’t automate this as much as we wanted, so we were missing out on upselling opportunities. Our senior management was keen that we fix this so the company would have the freedom to expand.”
Architects at Simplilearn knew that the solution lay in a decoupled event-streaming architecture based on Apache Kafka®. It could scale to handle trillions of messages a day and petabytes of data with the low latency that Simplilearn needed. But managing Kafka was not a task they wanted to take on.
“We’ve written our own queueing services before, so we know how resource-intensive they can be to manage,” says Saini.
Kafka represented an attractive off-the-shelf alternative but would require significant effort to manage the clusters. Knowing it wanted a managed service, the team compared offerings from Aiven and other vendors. “It was clear that Aiven for Apache Kafka had all the features, integrations and connectors we needed at the right price,” says Saini. “We chose to run it on Google Cloud, and it works seamlessly with our existing cross-cloud environment.”
The infrastructure team uses the Aiven GUI to add nodes to their Kafka cluster and create topics. “Working with Aiven meant we could focus on building our application,” says Wanchoo. “We didn’t have to manually provision Zookeeper or cluster nodes — or worry about any of the manual setup that follows. If we were managing Kafka in-house, we would need an extra DevOps engineer to look after it.”
The infrastructure team at Simplilearn has set alerts using Aiven integrators, so if the team sees unusual behavior, they can act fast. “Sometimes we can fix it, sometimes Aiven does, but the relationship is pretty seamless,” says Barua. “The customer success team at Aiven have gone out of their way to solve any challenges we’ve had, get us the right technical support and advise us on best practices and the plans we’re using. That help has only gotten better with time.”
Although the event-streaming technology is invisible to much of the Simplilearn community, the impact can be felt across the organization, from learners and teachers, through the development and infrastructure teams, right up to senior business leadership.
Those taking a course get a near-instant view of their progress. So do learner success managers, who can intervene if they notice a student is having difficulty and ensure that the learner progresses towards their goals.
Tech teams have an easier time managing a more automated, performant and resilient system so they can concentrate on activities that add value for their users. “Kafka is fault tolerant by design, and we’ve incorporated retry mechanisms, so the chance of failed writes is much lower,” says Saini. “It means users can carry on with their courses, and the heavy lifting is taken care of in the background without affecting the UX.”
“Fault tolerance also makes code releases easier,” adds Wanchoo. “If developers make a change that breaks a service that’s on Aiven, we’ll see that records are failing. We can fix the issue, make the release, and the failed records get sent to a retry queue to be run after the fix.”
Real-time, accurate data is also vital for the growth of Simplilearn as a business. It enables the company to offer more timely, tailored suggestions for learners, helping them find new courses to further their careers as well as retaining them as customers.
The success of the business has enabled Simplilearn to embark on new initiatives focusing on upskilling people for digital careers and landing the job they want.
Recent projects include a partnership with HerKey (formerly JobsForHer), India’s largest online platform for women’s careers. In 2022, the partnership trained 1,000 women in full-stack Java development, helping them get jobs in a vibrant IT sector and reducing the gender gap.
Another partnership with the Commonwealth is providing free digital upskilling courses for 10,000 students across the Caribbean, Africa and Asia through the Simplilearn SkillUp platform.
