alibaba/canal · error · CanalMetaManagerException

batchId:%d is not the firstly:%d

Error message

batchId:%d is not the firstly:%d

What it means

Thrown by ZooKeeperMetaManager during removePositionRange (ack/rollback path) when the given batchId exists in ZK but is not the minimum batchId. Like the memory manager, the ZK-backed meta enforces strict FIFO acknowledgement to avoid skipping events, but here the batch set is read from ZK child nodes of the batch-mark path.

Source

Thrown at meta/src/main/java/com/alibaba/otter/canal/meta/ZooKeeperMetaManager.java:201

    public PositionRange removeBatch(ClientIdentity clientIdentity, Long batchId) throws CanalMetaManagerException {
        String batchsPath = ZookeeperPathUtils.getBatchMarkPath(clientIdentity.getDestination(),
            clientIdentity.getClientId());
        List<String> nodes = zkClientx.getChildren(batchsPath);
        if (CollectionUtils.isEmpty(nodes)) {
            // 没有batch记录
            return null;
        }

        // 找到最小的Id
        ArrayList<Long> batchIds = new ArrayList<>(nodes.size());
        for (String batchIdString : nodes) {
            batchIds.add(Long.valueOf(batchIdString));
        }
        Long minBatchId = Collections.min(batchIds);
        if (!minBatchId.equals(batchId)) {
            // 检查一下提交的ack/rollback，必须按batchId分出去的顺序提交，否则容易出现丢数据
            throw new CanalMetaManagerException(String.format("batchId:%d is not the firstly:%d", batchId, minBatchId));
        }

        if (!batchIds.contains(batchId)) {
            // 不存在对应的batchId
            return null;
        }
        PositionRange positionRange = getBatch(clientIdentity, batchId);
        if (positionRange != null) {
            String path = ZookeeperPathUtils
                .getBatchMarkWithIdPath(clientIdentity.getDestination(), clientIdentity.getClientId(), batchId);
            zkClientx.delete(path);
        }

        return positionRange;
    }

    public PositionRange getBatch(ClientIdentity clientIdentity, Long batchId) throws CanalMetaManagerException {
        String path = ZookeeperPathUtils

View on GitHub (pinned to 87be50e876)

Solutions

  1. Ack batches in ascending batchId order; always resolve and ack the minimum outstanding first.
  2. Ensure each consumer uses a unique clientId so concurrent acks do not interleave on the same batch namespace.
  3. If stale nodes are corrupting order, clean the batch-mark ZK path for that destination/clientId (zkCli delete) as a last resort, after confirming no live consumer.
  4. Audit client code for logic that reorders batches (e.g. acking after a retry that already advanced the cursor).
Defensive patterns

Strategy: validation

Validate before calling

// list ZK batch nodes and confirm batchId is the min before ack
List<String> nodes = zkClient.getChildren(batchMarkPath);
long min = nodes.stream().mapToLong(Long::parseLong).min().orElseThrow();
if (min != batchId) throw new IllegalStateException("out-of-order ack; oldest=" + min);

Try / catch

try { client.ack(batchId); }
catch (CanalMetaManagerException e) {
    if (e.getMessage().startsWith("batchId:")) { /* reconcile: ack oldest or rollback */ }
    else throw e;
}

Prevention

When it happens

Trigger: ZooKeeperMetaManager processes an ack/rollback for batchId; it lists batch nodes, converts them to Long, computes Collections.min, and if that min != batchId it throws. Occurs on client.ack(batchId) / client.rollback(batchId) when an older batch node still exists.

Common situations: Out-of-order ack across clients sharing a clientId, a client that fetched multiple batches but acks a later one first, stale batch nodes left in ZK after a crash, or two Canal clients racing on the same clientId.

Related errors

AI-assisted analysis of alibaba/canal@87be50e876 (2026-08-14). Data as JSON: /api/errors/de0e687da07994ee. Report an issue: GitHub.