apache/pulsar · error · ManagedLedgerException

Timeout during mark-delete operation

Error message

Timeout during mark-delete operation

What it means

When the GCS offload driver is configured with gcsManagedLedgerOffloadServiceAccountKeyFile, buildCredentials reads that JSON key file; if an IOException occurs while reading it, the error is logged and rethrown as IllegalArgumentException. The credentials file is missing, unreadable, or otherwise failed I/O, so GCS authentication cannot be set up.

Source

Thrown at managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:1977

        final Result result = new Result();
        final CountDownLatch counter = new CountDownLatch(1);

        asyncMarkDelete(position, properties, new MarkDeleteCallback() {
            @Override
            public void markDeleteComplete(Object ctx) {
                counter.countDown();
            }

            @Override
            public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {
                result.exception = exception;
                counter.countDown();
            }
        }, null);

        if (!counter.await(ManagedLedgerImpl.AsyncOperationTimeoutSeconds, TimeUnit.SECONDS)) {
            throw new ManagedLedgerException("Timeout during mark-delete operation");
        }

        if (result.exception != null) {
            throw result.exception;
        }
    }

    @Override
    public void clearBacklog() throws InterruptedException, ManagedLedgerException {
        class Result {
            ManagedLedgerException exception = null;
        }

        final Result result = new Result();
        final CountDownLatch counter = new CountDownLatch(1);

        asyncClearBacklog(new ClearBacklogCallback() {
            @Override

View on GitHub (pinned to 820761864e)

Solutions

1. Fix gcsManagedLedgerOffloadServiceAccountKeyFile to an absolute path pointing at the existing service-account JSON key file.
2. Verify the broker process user can read the file (ls -l; check ownership/permissions, and that the K8s secret is mounted).
3. Alternatively use workload identity / application-default credentials and drop the key-file property.
4. Check the broker log for the 'file' attribute in this error entry to see exactly which path failed.

Example fix

// before: relative or wrong path
offloaderProperties.gcsManagedLedgerOffloadServiceAccountKeyFile=keys/gcs.json

// after: absolute, existing, readable path
offloaderProperties.gcsManagedLedgerOffloadServiceAccountKeyFile=/etc/pulsar/gcs/gcs-service-account.json

String keyFile = config.getConfigProperty("gcsManagedLedgerOffloadServiceAccountKeyFile");
java.io.File f = new java.io.File(keyFile);
if (keyFile == null || !f.isFile() || !f.canRead()) {
    throw new IllegalStateException("GCS key file missing or unreadable: " + keyFile);
}
new String(java.nio.file.Files.readAllBytes(f.toPath()), Charset.defaultCharset()); // ensures parseable read

try {
    blobStore = provider.getBlobStore(tieringConfig);
} catch (IllegalArgumentException e) {
    if (e.getCause() instanceof IOException) {
        log.error("GCS credentials file could not be read: {}", e.getCause().getMessage());
    }
    throw e;
}

When it happens

Trigger: Calling JCloudBlobStoreProvider.buildCredentials (via blob store construction/offload start) with provider GCS and gcsManagedLedgerOffloadServiceAccountKeyFile pointing to a nonexistent, permission-denied, or unreadable file path.

Common situations: Typo in the key-file path in broker.conf, key file deleted/moved after deployment, running the broker as a user lacking read permission on the mounted secret, Kubernetes secret not mounted or wrong mount path, relative path resolved against an unexpected working directory.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timeout during clear backlog operation
• Timeout during skip messages operation
• Could not open configuration file
• ioThreads must be >= 1
• Both ${S3_ID_FIELD} and ${S3_SECRET_FIELD} must be set when

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/493e46e298bd1630. Report an issue: GitHub.