Building Data Mesh at Scale: Transforming Real-Time Analytics at Netflix

In this talk, Sujay Jain will delve into the transformative journey of building a state-of-the-art Data Mesh at Netflix, a platform that processes trillions of events daily to power various business-critical applications. Attendees will gain insights into how Netflix leverages this distributed, real-time data movement and processing infrastructure to deliver personalized user experiences. The core of the discussion will be on how the Data Mesh at Netflix operates at scale with about 20,000 Apache Flink jobs and manages thousands of Kafka topics. Sujay will explore the challenges faced in developing such a robust system, particularly focusing on scaling, reliability, and the intricacies of managing high-throughput, low-latency data streams. Participants will learn about the innovative use of Streaming SQL, which Sujay led, to democratize stream processing, allowing platform users to perform complex data transformations with ease.