apache/pulsar · warning · org.apache.pulsar.broker.service.AlreadyRunningException

Offload already in progress

Error message

Offload already in progress

What it means

Like compaction, only one offload operation may run per topic; PersistentTopic tracks it with a currentOffload CompletableFuture. Calling triggerOffload while a previous offload is still in progress throws AlreadyRunningException('Offload already in progress'). The guard serializes storage offload work so two runs cannot offload the same ledgers concurrently.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:4975

                        public void offloadComplete(Position pos, Object ctx) {
                            Position impl = pos;
                            log.info()
                                    .attr("messageId", messageId)
                                    .log("Completed successfully offload operation at messageId");
                            promise.complete(new MessageIdImpl(impl.getLedgerId(), impl.getEntryId(), -1));
                        }

                        @Override
                        public void offloadFailed(ManagedLedgerException exception, Object ctx) {
                            log.warn()
                                    .attr("messageId", messageId)
                                    .exception(exception)
                                    .log("Failed offload operation at messageId");
                            promise.completeExceptionally(exception);
                        }
                    }, null);
        } else {
            throw new AlreadyRunningException("Offload already in progress");
        }
    }

    public synchronized OffloadProcessStatus offloadStatus() {
        if (!currentOffload.isDone()) {
            return OffloadProcessStatus.forStatus(LongRunningProcessStatus.Status.RUNNING);
        } else {
            try {
                if (currentOffload.join() == MessageId.earliest) {
                    return OffloadProcessStatus.forStatus(LongRunningProcessStatus.Status.NOT_RUN);
                } else {
                    return OffloadProcessStatus.forSuccess(currentOffload.join());
                }
            } catch (CancellationException | CompletionException e) {
                log.warn("Failed to offload");
                return OffloadProcessStatus.forError(e.getMessage());
            }
        }

View on GitHub (pinned to 820761864e)

Solutions

1. Check admin topics().offloadStatus() and trigger only when status is not RUNNING
2. Catch AlreadyRunningException and skip/back off, retrying after offloadStatus() reports completion
3. Throttle scheduled offload triggers based on measured offload duration; add per-topic locking
4. Speed up or investigate offload throughput (object storage latency, bandwidth limits) if it routinely overlaps

Example fix

// before
admin.topics().triggerOffload(topic, messageId); // throws if running
// after
if (!OffloadProcessStatus.Status.RUNNING.equals(admin.topics().offloadStatus(topic).status)) {
    admin.topics().triggerOffload(topic, messageId);
}

OffloadProcessStatus st = admin.topics().offloadStatus(topic);
if (st.status == OffloadProcessStatus.Status.RUNNING) {
    return; // skip trigger
}

boolean canTriggerOffload(OffloadProcessStatus s) {
    return s != null && s.status != OffloadProcessStatus.Status.RUNNING;
}

try {
    admin.topics().triggerOffload(topic, messageId);
} catch (PulsarAdminException e) {
    if (e.getCause() instanceof AlreadyRunningException) {
        log.info("Offload already running on {}", topic);
    } else throw e;
}

When it happens

Trigger: Calling admin topics().triggerOffload() (or internal offload APIs) while the topic's currentOffload future is not done — repeated manual triggers, overlapping scheduled offload jobs, or retry loops without checking offloadStatus().

Common situations: Automated offload scripts with intervals shorter than offload duration; operator re-clicking the offload trigger while a large-ledger offload runs; offloads slowed by slow object storage (S3/GCS) causing overlap of subsequent triggers.

Related errors

• The offloadPolicies must be specified for namespace offload.
• The driver is not supported, support value: ${supportedDrive
• The bucket must be specified for namespace offload.
• Compaction already in progress
• Timeout during managed ledger offload operation

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/bcf15cfea324a175. Report an issue: GitHub.