In this post, I try to explain all about Brokers in Kafka.
What is a Broker?
The basic component of Kafka Infrastructure is Broker.
Kafka Broker, Kafka Instance, Kafka Node all are same and it is used by producer to post message and consumer to get message. In simple words, Kafka Broker is a running instance of Kafka.
Multiple Brokers forms Kafka Cluster which is used for load balancing.
Lets see the technical details of broker's configuration
Broker configurations are in server.properties configuration file (Path: config/server.properties)
broker.id - it is the unique id of the broker. It can be manually defined and if undefined zookeeper will generate. To avoid conflict between zookeeper generated and user generated broker id's, broker ids generated from reserved.broker.max.id+1. On cluster environment cannot have two brokers with same id.
listeners - contains the list of all possible listeners(urls). We can specify hostname and port to bind to specific interface or 0.0.0.0 to bind to all interfaces or leave it empty to bind to default interfaces (eg: PLAINTEXT://0.0.0.0:port,SSL://:port)
advertised.listeners - is used by the external clients. Listeners are for the internal network and if any external client needa to communicate with brokers then an external ip/port is defined in advertised.listeners(0.0.0.0 is not valid for advertised.listeners). To know more about listener configuraions please refer rmoff's blog.
listener.securiry.protocol.map - this parameter is used to map security protocols for specific listeners. If we are using more than one port and if we need to use different security mapping ls for them, we can define them using this parameter. For example, if internal and external traffic can be seperated using this property as 'INTERNAL:PLAINTEXT, EXTERNAL:SSL'
num.network.threads - it is used to define the number of threads used by the broker to send/receive requests from/to network. The default value is 3 and should start from 1.
num.io.threads - it is used to define the number of threads used by broker to process requests including disk I/O. The default value is 8 and should start from 1.
log.dirs - It is used to define the log path for the Kafka broker.
num.partitions - This parameter defines the default number of partitions for a Kafka broker.
num.recovery.threads.per.data.dir - It defines the number of threads can be used for log recovery.
log.retention.hours - This parameter is used to define the age upto which the log file will be available for processing. After the defined hours it will be removed.
log.retention.bytes - It defines a size based retention policy.
log.segment.bytes - It defines the maximum size of a log file. Once the log file reached the defined size, a new log segment will be created.
log.retention.check.interval.ms - It defines the interval time at which the log segments will be checked for removal as per retention policies.
zookeeper.connect - Defines the Zoo keeper connection string for the broker.
zookeeper.connection.timeout.ms - It sets up the timeout in ms for the zookeeper connection.
I hope this post gave a basic knowledge on what is broker and what are all the configurations of server.properties file.
