apache/pulsar · error · ManagedLedgerException

Timeout during skip messages operation

Error message

Timeout during skip messages operation

What it means

Azure Blob offload builds Credentials(accountName, accountKey) from AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_ACCESS_KEY. This IllegalArgumentException is thrown when AZURE_STORAGE_ACCESS_KEY is empty or unset while the storage account name was present. Without the access key no shared-key authentication can be performed.

Source

Thrown at managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:2061

        final Result result = new Result();
        final CountDownLatch counter = new CountDownLatch(1);

        asyncSkipEntries(numEntriesToSkip, deletedEntries, new SkipEntriesCallback() {
            @Override
            public void skipEntriesComplete(Object ctx) {
                counter.countDown();
            }

            @Override
            public void skipEntriesFailed(ManagedLedgerException exception, Object ctx) {
                result.exception = exception;
                counter.countDown();
            }
        }, null);

        if (!counter.await(ManagedLedgerImpl.AsyncOperationTimeoutSeconds, TimeUnit.SECONDS)) {
            throw new ManagedLedgerException("Timeout during skip messages operation");
        }

        if (result.exception != null) {
            throw result.exception;
        }
    }

    @Override
    public void asyncSkipEntries(int numEntriesToSkip, IndividualDeletedEntries deletedEntries,
            final SkipEntriesCallback callback, Object ctx) {
        log.info().attr("numEntriesToSkip", numEntriesToSkip).log("Skipping entries");
        long numDeletedMessages = 0;
        if (deletedEntries == IndividualDeletedEntries.Exclude) {
            numDeletedMessages = getNumIndividualDeletedEntriesToSkip(numEntriesToSkip);
        }

        asyncMarkDelete(ledger.getPositionAfterN(markDeletePosition, numEntriesToSkip + numDeletedMessages,
                PositionBound.startExcluded), new MarkDeleteCallback() {

View on GitHub (pinned to 820761864e)

Solutions

1. Set AZURE_STORAGE_ACCESS_KEY to the storage account's key (Azure Portal -> Storage account -> Access keys).
2. Ensure both AZURE variables are injected together in the same deployment unit (Docker env, K8s envFrom/secret, systemd EnvironmentFile).
3. Verify the value is non-empty after shell/K8s expansion (quotes, $ escaping, secretRef names).
4. Consider migrating to Azure managed identity / token credential providers to avoid key handling.

Example fix

// before: account set, key missing
export AZURE_STORAGE_ACCOUNT=mystorageacct

// after: both required variables exported
export AZURE_STORAGE_ACCOUNT=mystorageacct
export AZURE_STORAGE_ACCESS_KEY=$(az storage account keys list -n mystorageacct --query '[0].value' -o tsv)

String key = System.getenv("AZURE_STORAGE_ACCESS_KEY");
if (key == null || key.isEmpty()) {
    throw new IllegalStateException("AZURE_STORAGE_ACCESS_KEY must be set for azureblob offload");
}
// optional sanity: Base64 decodable
java.util.Base64.getDecoder().decode(key);

try {
    provider.buildCredentials(tieringConfig);
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("access key")) {
        log.error("Set AZURE_STORAGE_ACCESS_KEY for the storage account named in AZURE_STORAGE_ACCOUNT");
    }
    throw e;
}

When it happens

Trigger: Calling buildCredentials for driver=azureblob when AZURE_STORAGE_ACCESS_KEY is missing/empty (account name check at the preceding line already passed).

Common situations: Only AZURE_STORAGE_ACCOUNT was exported; the secret containing the key was not injected in K8s; key was rotated and the env var left blank; quoting issues in docker/systemd dropping the variable.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timeout during clear backlog operation
• Timeout during mark-delete operation
• Both ${S3_ID_FIELD} and ${S3_SECRET_FIELD} must be set when
• Couldn't get the access key id.
• Couldn't get the access key secret.

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/8d5e2246b1010dd7. Report an issue: GitHub.