What causes "Timeout during managed ledger offload operation" in apache/pulsar?

Use asyncOffloadPrefix() instead of the sync offloadPrefix() and poll the returned future yourself

apache/pulsar · error · ManagedLedgerException

Timeout during managed ledger offload operation

Error message

Timeout during managed ledger offload operation

What it means

ManagedLedgerImpl.offloadPrefix() submits an async offload and blocks on the returned future with a bounded get(AsyncOperationTimeoutSeconds). If offloading does not complete in time, the TimeoutException is wrapped into this ManagedLedgerException. Offloading moves older ledgers to long-term storage (e.g. S3/GCS), which can be slow for large data.

Source

Thrown at managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:3643

    public Position offloadPrefix(Position pos) throws InterruptedException, ManagedLedgerException {
        CompletableFuture<Position> promise = new CompletableFuture<>();

        asyncOffloadPrefix(pos, new OffloadCallback() {
            @Override
            public void offloadComplete(Position offloadedTo, Object ctx) {
                promise.complete(offloadedTo);
            }

            @Override
            public void offloadFailed(ManagedLedgerException e, Object ctx) {
                promise.completeExceptionally(e);
            }
        }, null);

        try {
            return promise.get(AsyncOperationTimeoutSeconds, TimeUnit.SECONDS);
        } catch (TimeoutException te) {
            throw new ManagedLedgerException("Timeout during managed ledger offload operation");
        } catch (ExecutionException e) {
            log.error().attr("position", pos).exception(e.getCause()).log("Error offloading");
            throw ManagedLedgerException.getManagedLedgerException(e.getCause());
        }
    }

    @Override
    public void asyncOffloadPrefix(Position pos, OffloadCallback callback, Object ctx) {
        LedgerOffloader ledgerOffloader = config.getLedgerOffloader();
        if (ledgerOffloader != null && !ledgerOffloader.isAppendable()) {
            String msg = String.format("[%s] does not support offload", ledgerOffloader.getClass().getSimpleName());
            callback.offloadFailed(new ManagedLedgerException(msg), ctx);
            return;
        }
        Position requestOffloadTo = pos;
        if (!isValidPosition(requestOffloadTo)
                // Also consider the case where the last ledger is currently
                // empty. In this the passed position is not technically

View on GitHub (pinned to 820761864e)

Solutions

Use asyncOffloadPrefix() instead of the sync offloadPrefix() and poll the returned future yourself
Increase AsyncOperationTimeoutSeconds to accommodate the offloader's throughput for your data volume
Verify offloader configuration (bucket/endpoint/credentials) and object-store network performance
Reduce the offload threshold/size per invocation so each offload completes faster

Example fix

// before
managedLedger.offloadPrefix(position); // blocks, throws on timeout
// after
managedLedger.asyncOffloadPrefix(position)
    .orTimeout(10, TimeUnit.MINUTES)
    .thenAccept(res -> log.info("offloaded to {}", res));

Defensive patterns

Strategy: try-catch

Validate before calling

// check offloader reachable and backlog size
long bytes = ml.getEstimatedBacklogSize();
if (!offloaderHealthy() || bytes > MAX_OFFLOAD_BATCH) { offloadInChunks(); return; }

Try / catch

try {
    ml.offloadPrefix(pos);
} catch (ManagedLedgerException e) {
    log.warn("offload timed out; reduce batch or use asyncOffloadPrefix", e);
}

Prevention

Use asyncOffloadPrefix and manage the future yourself
Offload in smaller increments
Validate object-store credentials/endpoint before offloading
Raise AsyncOperationTimeoutSeconds for large data volumes

When it happens

Trigger: Calling offloadPrefix() where reading from BookKeeper plus writing to the offloader (S3, GCS, filesystem) exceeds AsyncOperationTimeoutSeconds — large ledgers, slow object-store bandwidth, or offloader misconfiguration (retries against unreachable bucket).

Common situations: Manual offload of a large backlog; automatic offload on a topic with a large unoffloaded tail; storage plugin credentials/bucket wrong so writes hang and retry; slow network to object storage.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/d5a3388778b9e434. Report an issue: GitHub.