Apache Kafka® is an event streaming platform. What does that mean?
Kafka combines three key capabilities so you can implement your use cases for event streaming end-to-end with a single battle-tested solution:
- To publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems.
- To store streams of events durably and reliably for as long as you want.
- To process streams of events as they occur or retrospectively.
- kafka connect
- confluent kafka
How does it work?
- Kafka is a distributed system consisting of servers and clients communicating via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers on-premise as well as in cloud environments.
🖥️ 1. Kafka Servers (Brokers)
📌 Purpose:
Kafka servers, also called brokers, are responsible for storing and managing the data streams (messages) in Kafka.
Main Roles:
- Receive messages from producers
- Store messages on disk in topics and partitions
- Serve messages to consumers
- Handle replication for fault tolerance
- Coordinate with ZooKeeper (or KRaft) for cluster metadata