Kafka - Topics

1. Introduction to Kafka Topics

In Kafka, a topic is a category or feed name to which records are sent. Topics are the fundamental abstraction in Kafka that represent the stream of data, and all Kafka records are organized into topics. Understanding how to configure and manage Kafka topics is crucial for building efficient and scalable data pipelines.

2. Kafka Topic Diagram Explanation

The diagram below visually represents the key components and configurations of Kafka topics, including partitions, replication, and data flow within a Kafka cluster. Understanding this diagram will help you grasp the core concepts of how Kafka organizes and manages data across distributed systems.

Kafka Topic Structure Diagram


Kafka Topic Diagram

2.1. Partitions

In Kafka, a topic is divided into partitions, which are the fundamental units of parallelism and scalability. Each partition is an ordered sequence of records, and Kafka guarantees the order of records within a partition. In the diagram, each partition is shown as a separate segment, indicating how data is distributed within a topic.


2.2. Replication

Kafka ensures fault tolerance and high availability by replicating partitions across multiple brokers. The diagram shows leader and follower replicas for each partition:


2.3. Data Flow and Fault Tolerance

The diagram illustrates the data flow within Kafka topics, showing how messages are produced to specific partitions and replicated across brokers. The replication process ensures that even if one broker fails, the data remains available through the follower replicas. This setup enhances Kafka's reliability and fault tolerance, making it suitable for critical applications where data loss is unacceptable.


2.4. Topic Configuration

The diagram also highlights key configuration aspects of Kafka topics, such as the number of partitions and the replication factor. These configurations are essential for tuning Kafka's performance and ensuring it meets the scalability and reliability requirements of your application.

By studying this diagram, you can gain a clearer understanding of how Kafka organizes data into topics, partitions, and replicas. This knowledge is crucial for designing effective Kafka-based data streaming architectures that are both scalable and resilient to failures.

3. Topic Configuration

Kafka topics can be configured with various settings that control their behavior, including the number of partitions, replication factor, and retention policies. These configurations are critical for optimizing the performance and reliability of your Kafka cluster.


3.1. Number of Partitions

Partitions are a fundamental unit of parallelism in Kafka. Configuring the number of partitions for a topic determines how the data is distributed and processed across consumers. The more partitions a topic has, the more consumers can read from it concurrently, which can increase throughput. 

// Example: Create a topic with a specific number of partitions using Confluent.Kafka.AdminClient in C#
var adminConfig = new AdminClientConfig { BootstrapServers = "localhost:9092" };
using var adminClient = new AdminClientBuilder(adminConfig).Build();
await adminClient.CreateTopicsAsync(new List<TopicSpecification> {
    new TopicSpecification { Name = "test-topic", NumPartitions = 3, ReplicationFactor = 2 }
});

3.2. Replication Factor

The replication factor determines how many copies of the data are stored across the Kafka cluster. A higher replication factor improves fault tolerance, ensuring that data is not lost if a broker fails. 

// Example: Set replication factor while creating a topic
var topicSpec = new TopicSpecification { Name = "test-topic", NumPartitions = 3, ReplicationFactor = 3 };
await adminClient.CreateTopicsAsync(new List<TopicSpecification> { topicSpec });

4. Partitioning in Kafka

Partitioning is a key feature of Kafka that allows for horizontal scaling of data processing. Each partition in a topic is an ordered sequence of records, and Kafka ensures that records with the same key are placed in the same partition, maintaining order within that partition.


4.1. Partitioning Strategy

Kafka uses a partitioning strategy to determine which partition a record should go to. By default, Kafka uses key-based partitioning, where the key is hashed to select a partition. Custom partitioning strategies can also be implemented for specific use cases. 

// Example: Producing messages with key-based partitioning
var producerConfig = new ProducerConfig { BootstrapServers = "localhost:9092" };
using var producer = new ProducerBuilder<string, string>(producerConfig).Build();
await producer.ProduceAsync("test-topic", new Message<string, string> { Key = "userId", Value = "userData" });

5. Replication in Kafka

Replication is the process of copying data across multiple brokers to ensure high availability and fault tolerance. Kafka replicates data across brokers in a way that allows it to continue operating even if some brokers fail.


5.1. Leader and Follower Replicas

In Kafka, each partition has one leader replica and one or more follower replicas. The leader replica handles all read and write requests, while the follower replicas replicate the leader's data. If the leader fails, one of the followers is automatically elected as the new leader. 

// Example: Checking the leader for a partition using Confluent.Kafka.AdminClient
var metadata = adminClient.GetMetadata("test-topic", TimeSpan.FromSeconds(10));
foreach (var partition in metadata.Topics[0].Partitions)
{
    Console.WriteLine($"Partition {partition.PartitionId}, Leader: {partition.Leader}");
}

6. Topic Retention Policies

Kafka allows configuring retention policies that determine how long records are retained in a topic. Retention can be based on time or size, allowing for flexible data management strategies.


6.1. Time-Based Retention

Time-based retention keeps records for a specified duration before they are deleted. This is useful for topics where only the most recent data is relevant. 

// Example: Setting time-based retention policy for a topic
await adminClient.AlterConfigsAsync(new List<ConfigResource> {
    new ConfigResource { Type = ResourceType.Topic, Name = "test-topic" }
}.ToDictionary(cr => cr, cr => new Dictionary<string, string> { ["retention.ms"] = "604800000" })); // 7 days

6.2. Size-Based Retention

Size-based retention deletes records once the topic reaches a certain size. This is useful for controlling the disk space used by Kafka. 

// Example: Setting size-based retention policy for a topic
await adminClient.AlterConfigsAsync(new List<ConfigResource> {
    new ConfigResource { Type = ResourceType.Topic, Name = "test-topic" }
}.ToDictionary(cr => cr, cr => new Dictionary<string, string> { ["retention.bytes"] = "104857600" })); // 100 MB

7. Topic Management

Managing Kafka topics includes tasks such as creating, deleting, and configuring topics. Kafka provides both command-line tools and APIs for managing topics programmatically.


7.1. Creating and Deleting Topics

Topics can be created and deleted using Kafka's AdminClient API or through command-line tools like `kafka-topics.sh`. 

// Example: Deleting a topic using Confluent.Kafka.AdminClient
await adminClient.DeleteTopicsAsync(new[] { "test-topic" });

7.2. Configuring Topics

Kafka topics can be reconfigured at runtime to adjust settings such as partition count, replication factor, and retention policies. This flexibility allows Kafka to adapt to changing workloads and requirements. 

// Example: Increasing the number of partitions for a topic
await adminClient.CreatePartitionsAsync(new Dictionary<string, int> { ["test-topic"] = 6 });

8. Monitoring and Managing Topics

Monitoring Kafka topics is essential for ensuring they operate efficiently and meet your application's needs. Kafka provides various metrics and tools for monitoring topics.


8.1. Topic Metrics

Important metrics to monitor for Kafka topics include partition size, message throughput, and replication status. These metrics help you understand how your topics are performing and whether they need reconfiguration.


8.2. Monitoring Tools

Various tools can be used to monitor Kafka topics, including:

9. Topic Evolution and Schema Management

As Kafka topics are used over time, the structure of the messages they carry might evolve. Managing this evolution, especially when it involves changing the data schema, is critical to maintaining compatibility across producers and consumers.


9.1. Schema Registry

A Schema Registry is a service that provides a centralized repository for schemas used by Kafka topics. It allows producers and consumers to agree on the structure of the data being exchanged, making it easier to handle schema changes over time. 

// Example: Registering a schema with Confluent's Schema Registry
var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig);
var schemaId = await schemaRegistry.RegisterSchemaAsync("test-topic-value", myAvroSchema);

9.2. Handling Schema Evolution

Schema evolution allows you to update the structure of your data without breaking existing consumers. Kafka supports backward and forward compatibility, ensuring that new and old versions of a schema can coexist. 

// Example: Handling schema evolution
// Ensure that new schema versions are backward compatible with older versions
schemaRegistry.RegisterSchemaAsync("test-topic-value", newSchema, SchemaCompatibility.BACKWARD);

10. Security and Access Control

Securing Kafka topics is crucial for ensuring that only authorized producers and consumers can read or write data. Kafka supports several security features, including authentication, authorization, and encryption.


10.1. Authentication

Kafka supports various authentication mechanisms, such as SSL/TLS, SASL, and Kerberos, to ensure that only trusted clients can connect to the Kafka cluster. 

// Example: Configuring SSL for a Kafka producer
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    SecurityProtocol = SecurityProtocol.Ssl,
    SslCaLocation = "/etc/kafka/secrets/ca-cert",
    SslCertificateLocation = "/etc/kafka/secrets/client-cert",
    SslKeyLocation = "/etc/kafka/secrets/client-key"
};

10.2. Authorization

Kafka's authorization features allow you to define ACLs (Access Control Lists) that specify which users or services have permissions to read from or write to specific topics. 

// Example: Defining an ACL to allow a specific user to produce messages to a topic
kafka-acls.sh --add --allow-principal User:producerUser --producer --topic test-topic --bootstrap-server localhost:9092

10.3. Encryption

Kafka supports encryption both at rest and in transit. At-rest encryption ensures that the data stored on disk is encrypted, while in-transit encryption protects data as it moves between producers, brokers, and consumers.

11. Best Practices for Kafka Topics

Following best practices when managing Kafka topics can significantly improve the performance, reliability, and security of your Kafka cluster.

12. Summary

Kafka topics are the backbone of Kafka's data streaming architecture. By understanding and applying the right configurations, partitioning strategies, replication settings, and security measures, you can ensure that your Kafka topics are optimized for performance, reliability, and security.