Wholesale distribution and fulfillment services company, Essendant, saves time and gains agility with Aiven's open source tool
Muralidhar Basani
|RSS FeedStaff Software Engineer at Aiven
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
We understand the power and the popularity of Apache Kafka®. Its open-source, highly available data streaming technology is perfectly suited to event-driven applications that need to transfer data at speed and at scale. It’s no surprise that instances of Kafka can be found at more than eight in ten of all Fortune 100 companies.
In fact, companies of all sizes depend on Kafka for data streaming, integration and analytics. That includes Essendant, which provides wholesale distribution and fulfillment services throughout the US. Every day, the company fills nearly 250,000 product lines, and ships 53,000 orders from 30 distribution centers across the US — with 99.9% accuracy. It works with more than 500 supplier partners, its inventory is vast and diverse, and its customers rely on it to extend their own reach and improve their supply chains. Not bad for a company that has just celebrated its centenary.
Rajesh Solanki is a Domain Architect, Middleware at Essendant. He explains why Kafka is so important to the business. “We have an incredibly data-intensive operation. We are in the business of moving a huge range of goods and supplies around the country. All that needs to be managed and recorded. And that means we need to generate, access and move a huge amount of accurate, timely data around our business too.”
Like many companies, Essendant has grown its use of Kafka over time. It initially started using Kafka to support the analytics around one specific product in its inventory. Then the order-entry team, which needed to store thousands of events a day, started using Kafka to ensure that the company’s applications had easy access to those events.
“Today we have two Kafka clusters,” says Solanki. “One is a three-node cluster for the shared infrastructure used by multiple teams who are just writing events. The other is a five-node cluster dedicated to order-entry and all the processes and applications that need to access it. However, we want to expand the business so my team has a goal to enable more real-time data pipelines to support this. The Kafka clusters will become an even more important element of our business.”
However, running Kafka in an enterprise setting is by no means straightforward. It is one of the reasons that so many organizations turn to Aiven to take on the burden and free up internal expertise to focus on value-generating work.
Additionally, governing Kafka in an enterprise setting is a complex business – and one that is often overlooked, at least initially. But as more and more organizations like Essendant adopt Kafka in their technology stack, they are discovering that inadequate governance and authorization of the hundreds of topics created can compromise the benefits of having high-performance data pipelines, streaming analytics, and data integration.
And so, to help make Apache Kafka governance easier for everyone, Aiven acquired Kafkawize back in September 2022 – which is now known as Klaw – to give users of open source Apache Kafka, like Essendant, a centralized governance layer on top of Kafka. Klaw provides data governance with enterprise-grade security and user-management functionality and solves problems organizations face – like user management and topic-level operations – when scaling the adoption of Kafka across several use-cases, with multiple responsible teams and approval pipelines. The aim is to help organizations make better decisions, enhance regulatory compliance and optimize the efficiency of their Kafka footprint.
Back to Essendant. “We were using Spring Boot to create and view topics in Kafka, so everything was manual,” Solanki says. “My team was also developing REST APIs and venturing into creating and modifying Access Control Lists (ACLs), so we started using a tool to help manage the Kafka environment. However, we quickly figured out that the support wasn’t at the level we needed and the pricing model wasn’t right. I thought, ‘let's investigate if there are any open-source options out there.’”
Essendant has an open-minded approach to its IT from the top down. “It’s important to give my team the freedom and flexibility to decide which technologies to implement. I’m a big supporter of tracing technical solutioning to business value. That's how Klaw came into the picture,” says Andy Kiss, SVP of IT and CIO, Essendant.
Solanki immediately found that Klaw is a straightforward tool to use. “It’s open source, so I downloaded it and quickly figured it out. It comes packaged and shipped as a standalone Spring Boot app to run as is, but we run it on a Tomcat web container which requires a little bit of tweaking – but only half a day.”
He continues: “We deployed Klaw in one of our POC environments. I showed it to my manager, a couple of architects and some of my team members. Everyone liked it, so we progressed with it and for the last three months we’ve been in production.”
Klaw gives Essendant a process- and web-based data governance toolkit, which various teams can use to submit requests for new topics and schemas in Kafka, as well as access authorizations and Kafka connectors. The self-service user interface is easy to use and prevents users from calling on the already stretched administrators.
“Klaw has really increased the agility of our teams and our decision-making capabilities,” says Solanki. “It streamlines colleagues' access to Kafka without adding workload to the central management team. That speeds things up. And it helps us make faster improvements to the data infrastructure underpinning the business – without giving up control over the changes.”
“For example, we have multiple MongoDB Kafka Connectors that send data between the database and Kafka in our development, testing, QA, pre-production and production environments. If there’s a missing event, developers can use Klaw to look at any issues, navigate easily and quickly, and see the status of the connector. They know what’s going on in a couple of minutes – without calling in my team to manually trouble-shoot. It gives us back about five or six hours every week.”
As Essendant’s developers have found, Klaw is really easy to use. “The users love Klaw! And giving them access with the right level of authentication is really easy too,” says Solanki. Klaw also supports the company’s governance and security posture. “Using Klaw means we’ve been able to give certain teams access to specific Kafka topics, and manage the members’ level of access as required. If an event needs to be consumed by two or three different business units, we can now control that. There are no roadblocks in getting access and viewing the content.”
Currently 15 developers are using Klaw at Essendant alongside the entire middleware team. And new plans are underway. “We are starting on a process of using Schema Registry,” explains Solanki. “In a few weeks, we’ll be carrying out everything via Klaw: requesting topics, ACLs, connectors and schemas in the Schema Registry directly from Klaw. It’s an exciting process, and one that is going to help us get even more out of Kafka.”
Further reading
