what’s the purpose of using a serializer before pushing data into a Kafka topic?

• Each programming language has its own way of representing data, which makes us want to use a way to represent data between different languages interchangeably.

  data = {"name": "Alice", "age": 25, "city": "New York"}
  serialized_data = json.dumps(data)
  
  => {"name": "Alice", "age": 25, "city": "New York"}
  
  

• Thrift and Protocol Buffers (Protobuf) are alternative serialization formats, mainly used for efficient data exchange in distributed systems.

• The same kafka topic can receive messages with different schemas

• watermark → window functions
• flink never holds to anydata, most of the time it acts as a passthrough layer

What does Flink provide that the Kafka consumer API doesn’t have:

1. Job Recovery/serializing/checkpointing
2. Late Arriving data
3. Windowing/group by/ Watermarking
4. Scalability/parallelism

• Session windows: is used when you want to do analytics on a specific user behavior
• sliding window: is used when you want to do some analytics regarding some min/max values in your data

Why Watermarking?

1. Late event handling: In real-world systems (e.g., Kafka, IoT devices, logs), events often arrive late due to network delays, processing bottlenecks, or retries. Watermarks define how much lateness is acceptable.