{"record":{"id":"493e46e298bd1630","repo":"apache/pulsar","slug":"timeout-during-mark-delete-operation","errorCode":null,"errorMessage":"Timeout during mark-delete operation","messagePattern":"Timeout during mark-delete operation","errorType":"exception","errorClass":"ManagedLedgerException","httpStatus":null,"severity":"error","filePath":"managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java","lineNumber":1977,"sourceCode":"\n        final Result result = new Result();\n        final CountDownLatch counter = new CountDownLatch(1);\n\n        asyncMarkDelete(position, properties, new MarkDeleteCallback() {\n            @Override\n            public void markDeleteComplete(Object ctx) {\n                counter.countDown();\n            }\n\n            @Override\n            public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {\n                result.exception = exception;\n                counter.countDown();\n            }\n        }, null);\n\n        if (!counter.await(ManagedLedgerImpl.AsyncOperationTimeoutSeconds, TimeUnit.SECONDS)) {\n            throw new ManagedLedgerException(\"Timeout during mark-delete operation\");\n        }\n\n        if (result.exception != null) {\n            throw result.exception;\n        }\n    }\n\n    @Override\n    public void clearBacklog() throws InterruptedException, ManagedLedgerException {\n        class Result {\n            ManagedLedgerException exception = null;\n        }\n\n        final Result result = new Result();\n        final CountDownLatch counter = new CountDownLatch(1);\n\n        asyncClearBacklog(new ClearBacklogCallback() {\n            @Override","sourceCodeStart":1959,"sourceCodeEnd":1995,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L1959-L1995","documentation":"When the GCS offload driver is configured with gcsManagedLedgerOffloadServiceAccountKeyFile, buildCredentials reads that JSON key file; if an IOException occurs while reading it, the error is logged and rethrown as IllegalArgumentException. The credentials file is missing, unreadable, or otherwise failed I/O, so GCS authentication cannot be set up.","triggerScenarios":"Calling JCloudBlobStoreProvider.buildCredentials (via blob store construction/offload start) with provider GCS and gcsManagedLedgerOffloadServiceAccountKeyFile pointing to a nonexistent, permission-denied, or unreadable file path.","commonSituations":"Typo in the key-file path in broker.conf, key file deleted/moved after deployment, running the broker as a user lacking read permission on the mounted secret, Kubernetes secret not mounted or wrong mount path, relative path resolved against an unexpected working directory.","solutions":["Fix gcsManagedLedgerOffloadServiceAccountKeyFile to an absolute path pointing at the existing service-account JSON key file.","Verify the broker process user can read the file (ls -l; check ownership/permissions, and that the K8s secret is mounted).","Alternatively use workload identity / application-default credentials and drop the key-file property.","Check the broker log for the 'file' attribute in this error entry to see exactly which path failed."],"exampleFix":"// before: relative or wrong path\noffloaderProperties.gcsManagedLedgerOffloadServiceAccountKeyFile=keys/gcs.json\n\n// after: absolute, existing, readable path\noffloaderProperties.gcsManagedLedgerOffloadServiceAccountKeyFile=/etc/pulsar/gcs/gcs-service-account.json","handlingStrategy":"validation","validationCode":"String keyFile = config.getConfigProperty(\"gcsManagedLedgerOffloadServiceAccountKeyFile\");\njava.io.File f = new java.io.File(keyFile);\nif (keyFile == null || !f.isFile() || !f.canRead()) {\n    throw new IllegalStateException(\"GCS key file missing or unreadable: \" + keyFile);\n}\nnew String(java.nio.file.Files.readAllBytes(f.toPath()), Charset.defaultCharset()); // ensures parseable read","typeGuard":null,"tryCatchPattern":"try {\n    blobStore = provider.getBlobStore(tieringConfig);\n} catch (IllegalArgumentException e) {\n    if (e.getCause() instanceof IOException) {\n        log.error(\"GCS credentials file could not be read: {}\", e.getCause().getMessage());\n    }\n    throw e;\n}","preventionTips":["Use absolute paths for the service account key file.","In Kubernetes, verify the secret volume is mounted and readable by the pulsar user before broker start.","Add a startup health check that opens the key file when GCS offload is enabled.","Prefer workload identity / application-default credentials over key files."],"tags":["gcs","credentials","configuration","io"],"backgroundTag":"credentials-file-unreadable","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}