哪些网站容易做,盛大游戏优化大师,网站链接推广怎么赚钱,网站建设的基本技术文章目录 Kafka中涉及到的术语Kafka镜像选择Kafka UI镜像选择Docker Compose文件Kafka配置项说明KRaft vs Zookeeper和KRaft有关的配置关于Controller和Broker的概念解释Listener的各种配置 Kafka UI配置项说明 测试Kafka集群Docker Compose示例配置 Kafka中涉及到的术语
对于… 文章目录 Kafka中涉及到的术语Kafka镜像选择Kafka UI镜像选择Docker Compose文件Kafka配置项说明KRaft vs Zookeeper和KRaft有关的配置关于Controller和Broker的概念解释Listener的各种配置 Kafka UI配置项说明 测试Kafka集群Docker Compose示例配置 Kafka中涉及到的术语
对于Kafka中经常用到的术语可参考confluent的官方文档这里不再赘述。
Kafka镜像选择
镜像选择Docker Hub上使用最多的bitnami Kafka主要注意的点是环境变量和Kafka配置的映射关系 Additionally, any environment variable beginning with KAFKA_CFG_ will be mapped to its corresponding Apache Kafka key. For example, use KAFKA_CFG_BACKGROUND_THREADS in order to set background.threads or KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE in order to configure auto.create.topics.enable 还有就是当使用任何来自于bitnami的镜像如何遇到了问题想查看日志可以将镜像的Debug日志打开通过环境变量
BITNAMI_DEBUGtrue控制
由于Docker Hub的说明字数限制可以在Github上查看完整文档
Kafka UI镜像选择
对于Kafka的UI工具没有仔细调查原因是在使用初期阶段还不知道对于Kafka的监控和管理的痛点在哪所以先用起来再说。
Kafka UI官方Github地址 文档网址 Compose examples 下面可以找打很多Kafak ui的compose文件示例不仅对UI的配置很有帮助而且对刚入门Kafka的同学也提供了非常好的示例包括Kraft模式的Kafka集群等。
其他配置则阅读官方文档即可。
Docker Compose文件
version: 3
services:kafka:image: bitnami/kafka:latestcontainer_name: kafkaports:- 9092:9092- 9093:9093volumes:- type: volumesource: kafka_standalone_datatarget: /bitnami/kafkaread_only: falseenvironment:- BITNAMI_DEBUGyes# 启用KRaft模式必须设置下面三个属性- KAFKA_CFG_NODE_ID1- KAFKA_CFG_PROCESS_ROLESbroker,controller- KAFKA_CFG_CONTROLLER_LISTENER_NAMESCONTROLLER# broker id- KAFKA_BROKER_ID1# listener的各种配置- KAFKA_CFG_LISTENERSCONTROLLER://:9094,BROKER://:9092,EXTERNAL://:9093- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAPCONTROLLER:PLAINTEXT,BROKER:PLAINTEXT,EXTERNAL:PLAINTEXT# 注意EXTERNAL配置的是当前Docker所在的主机地址,BROKER可以使用Docker内部的网络地址即可- KAFKA_CFG_ADVERTISED_LISTENERSBROKER://kafka:9092,EXTERNAL://192.168.0.101:9093# 内部各个broker之间通信用的listener- KAFKA_CFG_INTER_BROKER_LISTENER_NAMEBROKER# 用来进行选举的Controller服务器如果有多个Controller则都需要写上这里本机- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS1127.0.0.1:9094- ALLOW_PLAINTEXT_LISTENERyeskafka-ui:container_name: kafka-uiimage: provectuslabs/kafka-ui:latestports:- 9095:8080depends_on:- kafkaenvironment:KAFKA_CLUSTERS_0_NAME: kafka-stand-aloneKAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092KAFKA_CLUSTERS_0_METRICS_PORT: 9997SERVER_SERVLET_CONTEXT_PATH: /kafkauiAUTH_TYPE: LOGIN_FORMSPRING_SECURITY_USER_NAME: adminSPRING_SECURITY_USER_PASSWORD: kafkauipasswordDYNAMIC_CONFIG_ENABLED: true
volumes:kafka_standalone_data:driver: localKafka配置项说明
KRaft vs Zookeeper
这里我们的配置是选择的是KRaft因为Kafka官方已经计划在Kafak中移除Zookeeper。至于为什么要移除confluentinc官方写了很多文章这里不再一一列举在Google上一搜就一大堆
KRaft site:confluent.io下面是几篇文章
why move to kraftWhy ZooKeeper Was Replaced with KRaftApache Kafka Without ZooKeeperGetting Started with the KRaft Protocol
和KRaft有关的配置
node.id The node ID associated with the roles this process is playing when process.roles is non-empty. This is required configuration when running in KRaft mode. porcess.roles The roles that this process plays: ‘broker’, ‘controller’, or ‘broker,controller’ if it is both. This configuration is only applicable for clusters in KRaft (Kafka Raft) mode (instead of ZooKeeper). Leave this config undefined or empty for Zookeeper clusters controller.listener.names A comma-separated list of the names of the listeners used by the controller. This is required if running in KRaft mode
关于Controller和Broker的概念解释
一句话解释
Controller负责协调Broker详细解释可见Kafak权威指南的第5章该书可在Apache Kafak官网 Get Started Books 中找到免费下载 To summarize, Kafka uses Zookeeper’s ephemeral node feature to elect a controller and to notify the controller when nodes join and leave the cluster. The controller is responsible for electing leaders among the partitions and replicas whenever it notices nodes join and leave the cluster. The controller uses the epoch number to prevent a “split brain” scenario where two nodes believe each is the current controller. Broker负责处理生产者生产消息的请求、存储消息、消费者消费消息的请求。 A single Kafka server is called a broker. The broker receives messages from producers, assigns offsets to them, and commits the messages to storage on disk. It also services consumers, responding to fetch requests for partitions and responding with the mes‐ sages that have been committed to disk 来自Kafka权威指南第1章Enter Kafka Broker And Clusters Listener的各种配置
当时看官方文档的时候这部分一直被搞得晕头转向直到看到这篇关于Kfaka的Listener文章才彻底明白了各种listener强烈推荐读一下这篇文章。
listener可分为3种
用来选举Controller的listenerkafka集群内部各broker节点通信的listener外部客户端例如Java Client连接Kafka
了解了3中controller结合上面的这篇文章Apache Kafka官方文档的配置说明配置listener就变得很容易了。
Kafka UI配置项说明
对于UI配置项没什么特别要说的这里只是提一下注意这里的docker-compose.yml中environment的写法和上面的Kafka镜像中environment的写法不同这是两种不同的写法 详细文档见Docker Compose文档规范中environment章节
测试
使用上述docker-compose.yml文件启动
docker-compose -f docker-compose.yml up -d在本地浏览器打开
http://localhost:9095/kafkaui/auth输入用户名、密码进入UI界面
Kafka集群Docker Compose示例配置
本篇文章重点在于搭建单机版Kfaka环境集群不在文章讨论范围之内如果对集群配置感兴趣和有需要的同学这里仅给出几个示例的Github仓库仅供参考并且全部使用KRaft而不是Zookeeper
我个人的理解最小的集群应该是3个controller3个broker, Kafka关于KRaft模式下Controller的部署建议
来自bitnami的Kafka集群示例三个节点皆为controller和broker。来自confluentic的Kafka集群配置我个人倾向于这个配置毕竟confluentic是Kafka商业化的公司其创始人来自LinkedIn。这里是四个节点一个controller和3个broker。来自Github的Kafka-In-Box使用4个节点。
