apache/pulsar · warning · org.apache.pulsar.broker.service.AlreadyRunningException

Offload already in progress

Error message

Offload already in progress

What it means

Like compaction, only one offload operation may run per topic; PersistentTopic tracks it with a currentOffload CompletableFuture. Calling triggerOffload while a previous offload is still in progress throws AlreadyRunningException('Offload already in progress'). The guard serializes storage offload work so two runs cannot offload the same ledgers concurrently.

Solutions

1. Check admin topics().offloadStatus() and trigger only when status is not RUNNING
2. Catch AlreadyRunningException and skip/back off, retrying after offloadStatus() reports completion
3. Throttle scheduled offload triggers based on measured offload duration; add per-topic locking
4. Speed up or investigate offload throughput (object storage latency, bandwidth limits) if it routinely overlaps

Example fix

// before
admin.topics().triggerOffload(topic, messageId); // throws if running
// after
if (!OffloadProcessStatus.Status.RUNNING.equals(admin.topics().offloadStatus(topic).status)) {
    admin.topics().triggerOffload(topic, messageId);
}

OffloadProcessStatus st = admin.topics().offloadStatus(topic);
if (st.status == OffloadProcessStatus.Status.RUNNING) {
    return; // skip trigger
}

boolean canTriggerOffload(OffloadProcessStatus s) {
    return s != null && s.status != OffloadProcessStatus.Status.RUNNING;
}

try {
    admin.topics().triggerOffload(topic, messageId);
} catch (PulsarAdminException e) {
    if (e.getCause() instanceof AlreadyRunningException) {
        log.info("Offload already running on {}", topic);
    } else throw e;
}

When it happens

Trigger: Calling admin topics().triggerOffload() (or internal offload APIs) while the topic's currentOffload future is not done — repeated manual triggers, overlapping scheduled offload jobs, or retry loops without checking offloadStatus().

Common situations: Automated offload scripts with intervals shorter than offload duration; operator re-clicking the offload trigger while a large-ledger offload runs; offloads slowed by slow object storage (S3/GCS) causing overlap of subsequent triggers.

Related errors

• Compaction already in progress
• The bucket must be specified for namespace offload.
• The driver is not supported, support value
• The offloadPolicies must be specified for namespace offload.
• Another partition exists for

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/bcf15cfea324a175. Report an issue: GitHub.

Appendix: source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:4975

                        public void offloadComplete(Position pos, Object ctx) {
                            Position impl = pos;
                            log.info()
                                    .attr("messageId", messageId)
                                    .log("Completed successfully offload operation at messageId");
                            promise.complete(new MessageIdImpl(impl.getLedgerId(), impl.getEntryId(), -1));
                        }

                        @Override
                        public void offloadFailed(ManagedLedgerException exception, Object ctx) {
                            log.warn()
                                    .attr("messageId", messageId)
                                    .exception(exception)
                                    .log("Failed offload operation at messageId");
                            promise.completeExceptionally(exception);
                        }
                    }, null);
        } else {
            throw new AlreadyRunningException("Offload already in progress");
        }
    }

    public synchronized OffloadProcessStatus offloadStatus() {
        if (!currentOffload.isDone()) {
            return OffloadProcessStatus.forStatus(LongRunningProcessStatus.Status.RUNNING);
        } else {
            try {
                if (currentOffload.join() == MessageId.earliest) {
                    return OffloadProcessStatus.forStatus(LongRunningProcessStatus.Status.NOT_RUN);
                } else {
                    return OffloadProcessStatus.forSuccess(currentOffload.join());
                }
            } catch (CancellationException | CompletionException e) {
                log.warn("Failed to offload");
                return OffloadProcessStatus.forError(e.getMessage());
            }
        }

View on GitHub (pinned to 820761864e)