Building a Real-Time Analytics Database • Tim Berglund • GOTO 2023

This presentation was recorded at GOTO Chicago 2023. #GOTOcon #GOTOchgo Tim Berglund - VP DevRel at StarTree & Author of “Gradle Beyond the Basics“ @tlberglund @startreefilms ORIGINAL TALK TITLE Building a Real-Time Analytics Database: A ’Choose Your Own Adventure’ Journey RESOURCES Tim ABSTRACT Have you ever stopped to think about how to build a database? The thing is, there isn’t just one way, as we can see by the massive number of data infrastructure options we have to choose from. It’s a nonstop series of tradeoffs, each motivated by the constraints the database wants to satisfy. An in-memory transactional database would be one thing. A general-purpose, single-server relational database would be another. A low-latency, horizontally scalable analytics database would journey we’re going to take. In this talk, we’ll start by picking a data model, make decisions about serialization and storage, choose indexing strategies, pick a query language, and figure out how to scale, eventually ending up with something that looks remarkably like Apache Pinot, a real-time analytics database. Pinot was built on a journey like this, always optimized for ultra low-latency, user-facing analytics at scale. In the real world, Pinot is used by applications like LinkedIn and UberEats to expose the state of the system not just to internal decision-makers, but to the users of the system itself, including all of us people who consumers of analytical queries. By focusing on the internals of Pinot and the tradeoffs made along the way to build a database of its kind, we’ll see how it enables a new class of applications that every user of a system into a decision maker. [...] Download slides and read the full abstract here: RECOMMENDED BOOKS Tim Berglund • Gradle Beyond the Basics • Tim Berglund & Matthew McCullough • Building and Testing with Gradle • Mark Needham • Building Real-Time Analytics Systems • Gwen Shapira, Todd Palino, Rajini Sivaram & Krit Petty • Kafka: The Definitive Guide • Adi Polak • Scaling Machine Learning with Spark • #ApachePinot #Analytics #RealTime #RealTimeAnalytics #TimBerglund #StarTree #StarTreeCloud #Cloud #ApachePinotTutorial #ApachePinotTraining #OLAP #OLTP #LowLatency #ApacheZooKeeper #ApacheHelix #Hadoop #ApacheSpark Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at Sign up for updates and specials at SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.