How to build data analytics pipelines faster than your morning commute
What is infrastructure? You may think of public transport and utilities, or perhaps networking and databases. All of those are infrastructure, and they actually have quite a bit in common. If words like “expensive”, “inconvenient” and “too slow” now come to mind, these are sad truths about buses, but your analytics pipelines could be running to a better schedule! Data is arriving faster and in greater quantities than ever before, from everywhere, all the time—we simply cannot afford to wait.
In this session, we will use live data feeds from Helsinki’s transport network to show how a complete open source data pipeline—from collection to insights—can be created in a matter of minutes using infrastructure-as-code and visualization tools you’re already familiar with. If you’re a software developer, platform engineer, data engineer, or analyst, we hope you’ll take away some tips and tricks to relieve the toil and frustration of your data infrastructure. (But sorry, we can’t fix the buses and trains…)
Related resources
Dave Beech
Staff Solution Architect, Aiven
Dave is a Solution Architect at Aiven and is passionate about data, tech, and espresso. He enjoys writing code (certainly a lot more than writing about himself) and has contributed to several open source projects in the big data ecosystem, including Apache Kafka, Avro, HBase and Crunch. Nowadays, Dave does more talking than typing, having spent the best part of the last 8 years as a technical architect and consultant, traveling the world (remember that?), and helping organizations of all shapes and sizes with their data problems.