KubeDay Japan 2024

Building a database and processing pipeline for petabyte-scale data can be challenging. Kubernetes can be a good choice, as it offers a highly available and resilient platform in a cost-optimized manner. However, managing data efficiently still requires careful design. Ensuring scalability often needs data sharding and sophisticated mechanisms for reading and writing data in parallel across a distributed system. If you're using Object Storage for cost efficiency, a distributed cache system using an appropriate data structure is a must-have. Our time-series database, which serves all the engineers in our company, handles over a billion metrics. The size of the stored data is likely to reach the petabyte scale. In this session, we'll explore how we utilized Object Storage, Kubernetes, and Envoy to construct an infinitely scalable time-series database. By the end, you will understand how to build an effective and high-performing distributed data processing system on Kubernetes.