alibaba/canal · error · CanalServerException

clientId:%s has last batch:[%s] isn't ack , maybe loss data

Error message

clientId:%s has last batch:[%s] isn't ack , maybe loss data

What it means

Thrown by CanalServerWithEmbedded.get() when MetaManager.getLastestBatch(clientIdentity) returns a non-null PositionRange. A non-null result means the client already holds an outstanding previously-fetched batch that was never acknowledged (ack) or rolled back (rollback). To prevent data loss the server refuses to hand out a new batch until the previous one is resolved.

Source

Thrown at server/src/main/java/com/alibaba/otter/canal/server/embedded/CanalServerWithEmbedded.java:251

     * b. 如果timeout不为null
     *    1. timeout为0，则采用get阻塞方式，获取数据，不设置超时，直到有足够的batchSize数据才返回
     *    2. timeout不为0，则采用get+timeout方式，获取数据，超时还没有batchSize足够的数据，有多少返回多少
     * 
     * 注意： meta获取和数据的获取需要保证顺序性，优先拿到meta的，一定也会是优先拿到数据，所以需要加同步. (不能出现先拿到meta，拿到第二批数据，这样就会导致数据顺序性出现问题)
     * </pre>
     */
    @Override
    public Message get(ClientIdentity clientIdentity, int batchSize, Long timeout, TimeUnit unit)
                                                                                                 throws CanalServerException {
        checkStart(clientIdentity.getDestination());
        checkSubscribe(clientIdentity);
        CanalInstance canalInstance = canalInstances.get(clientIdentity.getDestination());
        synchronized (canalInstance) {
            // 获取到流式数据中的最后一批获取的位置
            PositionRange<LogPosition> positionRanges = canalInstance.getMetaManager().getLastestBatch(clientIdentity);

            if (positionRanges != null) {
                throw new CanalServerException(String.format("clientId:%s has last batch:[%s] isn't ack , maybe loss data",
                    clientIdentity.getClientId(),
                    positionRanges));
            }

            Events<Event> events = null;
            Position start = canalInstance.getMetaManager().getCursor(clientIdentity);
            events = getEvents(canalInstance.getEventStore(), start, batchSize, timeout, unit);

            if (CollectionUtils.isEmpty(events.getEvents())) {
                logger.debug("get successfully, clientId:{} batchSize:{} but result is null",
                    clientIdentity.getClientId(),
                    batchSize);
                return new Message(-1, true, new ArrayList()); // 返回空包，避免生成batchId，浪费性能
            } else {
                // 记录到流式信息
                Long batchId = canalInstance.getMetaManager().addBatch(clientIdentity, events.getPositionRange());
                boolean raw = isRaw(canalInstance.getEventStore());
                List entrys = null;

View on GitHub (pinned to 87be50e876)

Solutions

  1. Ack or rollback the previously delivered batch (use the batchId from the last Message) before calling get() again.
  2. On client startup, call rollback(clientIdentity) to clear any stale outstanding batch left from a prior run.
  3. If the batch is legitimately lost, call rollback to reset the cursor to the last acked position, then resume get().
  4. Review client code to guarantee ack() is always invoked in a finally block after successful processing.

Example fix

// before: fetch without resolving prior batch
Message msg = server.get(clientId, 1000, 1L, TimeUnit.SECONDS);
process(msg);
server.ack(clientId, msg.getId());
// after: clear stale batch on connect, ack in finally
server.rollback(clientId); // clear any pending batch at startup
Message msg = server.get(clientId, 1000, 1L, TimeUnit.SECONDS);
try {
    process(msg);
    server.ack(clientId, msg.getId());
} catch (Exception e) {
    server.rollback(clientId, msg.getId());
}
Defensive patterns

Strategy: validation

Validate before calling

// before calling get(), ensure no outstanding batch is pending
PositionRange last = metaManager.getLastestBatch(clientIdentity);
if (last != null) {
    server.rollback(clientIdentity); // clear the stale batch first
}

Try / catch

try {
    Message msg = server.get(clientId, batchSize, timeout, unit);
} catch (CanalServerException e) {
    if (e.getMessage().contains("isn't ack")) {
        server.rollback(clientId); // resolve the pending batch then retry once
        return server.get(clientId, batchSize, timeout, unit);
    }
    throw e;
}

Prevention

When it happens

Trigger: Calling embeddedServer.get(clientIdentity, batchSize, timeout, unit) after a prior get() whose returned Message was neither acked nor rolled back. The MetaManager (memory or zookeeper backed) still records the last batch as pending.

Common situations: Client crashed or was killed mid-batch without acking; client bug that consumes get() but only conditionally calls ack(); long processing time where the prior batch is still in-flight; restart of the client while the server-side batch metadata persists in ZooKeeper.

Related errors

AI-assisted analysis of alibaba/canal@87be50e876 (2026-08-14). Data as JSON: /api/errors/83c80c669c99e4f5. Report an issue: GitHub.