Skip to main content
Tutorial series

Build a real-time data pipeline

This tutorial series explains the full process of building a real-time data pipeline based on KakaoCloud Advanced Managed Kafka.
It follows a single flow from message ingestion to data loading, metadata management, and query analysis, with each step written for a real operational environment.

Overall data flow: Advanced Managed Kafka -> Object Storage -> Data Catalog -> Data Query

architect Real-time data pipeline architecture

Tutorial structure​

The real-time data pipeline tutorial series consists of the following steps.

  1. Message processing through Kafka: Explains how to build a Kafka environment and send and receive messages.
  2. Load Kafka data into Object Storage: Shows how to load Kafka messages into KakaoCloud Object Storage and configure the storage step of a real-time data pipeline.
  3. Analyze Kafka messages using Data Catalog and Data Query: Explains how to register Kafka data stored in Object Storage with Data Catalog and query it through Data Query.