Hacking Limbo

Reading / Coding / Hacking

Kafka Partition Reassignment

第一次操作 topic partition 的迁移,有点紧张,实际过程很简单,而且迁移也很快(可能是因为数据不多)。要吐槽的是 Kafka 官方文档里对 partition / replica / broker 的编号规则有点混乱,一会是 0-based, 一会又变成 1-based 🙄. 具体操作过程如下:

  1. 运行 kafka-topics.sh --zookeeper zk:2181 --describe --topic <topic> 确认 topic partition 分布情况(集群操作只支持 zookeeper 定位,不能传 bootstrap server 地址),可以看到这个 topic 的 partition / replica 数量和所在节点。

  2. 构造 reassignment.json, 内容是:

    {
      "version": 1,
      "partitions": [
        {
          "topic": "<topic>",
          "partition": 0,
          "replicas": [2, 3]
        }
      ]
    }
  3. 运行 kafka-reassign-partitions.sh --zookeeper zk:2181 --reassignment-json-file reassignment.json —verify 看看 JSON 有没有写对──写对了的话会看到报错(是的)ERROR: Assigned replicas (1) don't match the list of replicas for reassignment (2,3) for partition [<topic>,0].

  4. 运行 kafka-reassign-partitions.sh --zookeeper zk:2181 --reassignment-json-file reassignment.json —execute 发起变更,然后再用 —verify 确认是否生效。

从官方文档里抄几个名词解释:

Replicas is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.

Isr is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

对于 PartitionCount == 1 && ReplicationFactor == 1 的 topic partition, 在 kafka-topics.sh --describe 输出里看到的 ReplicasIsr 应该只有(相同的)一个 broker 编号(假定 replica 和 broker 编号是一一对应的)。如果想列出只在 broker-1 上才有的 topics, 可以用命令 kafka-topics.sh --zookeeper zk:2181 --describe | grep -E 'Isr: 1$' 来过滤。

参考:

  1. Step 6: Setting up a multi-broker cluster
  2. Kafka Replication - Replica placements