{"record":{"id":"d5a3388778b9e434","repo":"apache/pulsar","slug":"timeout-during-managed-ledger-offload-operation","errorCode":null,"errorMessage":"Timeout during managed ledger offload operation","messagePattern":"Timeout during managed ledger offload operation","errorType":"exception","errorClass":"ManagedLedgerException","httpStatus":null,"severity":"error","filePath":"managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java","lineNumber":3643,"sourceCode":"    public Position offloadPrefix(Position pos) throws InterruptedException, ManagedLedgerException {\n        CompletableFuture<Position> promise = new CompletableFuture<>();\n\n        asyncOffloadPrefix(pos, new OffloadCallback() {\n            @Override\n            public void offloadComplete(Position offloadedTo, Object ctx) {\n                promise.complete(offloadedTo);\n            }\n\n            @Override\n            public void offloadFailed(ManagedLedgerException e, Object ctx) {\n                promise.completeExceptionally(e);\n            }\n        }, null);\n\n        try {\n            return promise.get(AsyncOperationTimeoutSeconds, TimeUnit.SECONDS);\n        } catch (TimeoutException te) {\n            throw new ManagedLedgerException(\"Timeout during managed ledger offload operation\");\n        } catch (ExecutionException e) {\n            log.error().attr(\"position\", pos).exception(e.getCause()).log(\"Error offloading\");\n            throw ManagedLedgerException.getManagedLedgerException(e.getCause());\n        }\n    }\n\n    @Override\n    public void asyncOffloadPrefix(Position pos, OffloadCallback callback, Object ctx) {\n        LedgerOffloader ledgerOffloader = config.getLedgerOffloader();\n        if (ledgerOffloader != null && !ledgerOffloader.isAppendable()) {\n            String msg = String.format(\"[%s] does not support offload\", ledgerOffloader.getClass().getSimpleName());\n            callback.offloadFailed(new ManagedLedgerException(msg), ctx);\n            return;\n        }\n        Position requestOffloadTo = pos;\n        if (!isValidPosition(requestOffloadTo)\n                // Also consider the case where the last ledger is currently\n                // empty. In this the passed position is not technically","sourceCodeStart":3625,"sourceCodeEnd":3661,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L3625-L3661","documentation":"ManagedLedgerImpl.offloadPrefix() submits an async offload and blocks on the returned future with a bounded get(AsyncOperationTimeoutSeconds). If offloading does not complete in time, the TimeoutException is wrapped into this ManagedLedgerException. Offloading moves older ledgers to long-term storage (e.g. S3/GCS), which can be slow for large data.","triggerScenarios":"Calling offloadPrefix() where reading from BookKeeper plus writing to the offloader (S3, GCS, filesystem) exceeds AsyncOperationTimeoutSeconds — large ledgers, slow object-store bandwidth, or offloader misconfiguration (retries against unreachable bucket).","commonSituations":"Manual offload of a large backlog; automatic offload on a topic with a large unoffloaded tail; storage plugin credentials/bucket wrong so writes hang and retry; slow network to object storage.","solutions":["Use asyncOffloadPrefix() instead of the sync offloadPrefix() and poll the returned future yourself","Increase AsyncOperationTimeoutSeconds to accommodate the offloader's throughput for your data volume","Verify offloader configuration (bucket/endpoint/credentials) and object-store network performance","Reduce the offload threshold/size per invocation so each offload completes faster"],"exampleFix":"// before\nmanagedLedger.offloadPrefix(position); // blocks, throws on timeout\n// after\nmanagedLedger.asyncOffloadPrefix(position)\n    .orTimeout(10, TimeUnit.MINUTES)\n    .thenAccept(res -> log.info(\"offloaded to {}\", res));","handlingStrategy":"try-catch","validationCode":"// check offloader reachable and backlog size\nlong bytes = ml.getEstimatedBacklogSize();\nif (!offloaderHealthy() || bytes > MAX_OFFLOAD_BATCH) { offloadInChunks(); return; }","typeGuard":null,"tryCatchPattern":"try {\n    ml.offloadPrefix(pos);\n} catch (ManagedLedgerException e) {\n    log.warn(\"offload timed out; reduce batch or use asyncOffloadPrefix\", e);\n}","preventionTips":["Use asyncOffloadPrefix and manage the future yourself","Offload in smaller increments","Validate object-store credentials/endpoint before offloading","Raise AsyncOperationTimeoutSeconds for large data volumes"],"tags":["timeout","offload","managed-ledger","storage"],"backgroundTag":"async-operation-timeout","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}