apache/pulsar · error · IllegalArgumentException

Both ${S3_ID_FIELD} and ${S3_SECRET_FIELD} must be set when

Error message

Both ${S3_ID_FIELD} and ${S3_SECRET_FIELD} must be set when providing offload credentials in the configuration

What it means

Thrown when S3-style offload credentials are supplied via configuration but only one of the two required keys is present: both the access-key-id field (S3_ID_FIELD, e.g. s3ManagedLedgerOffloadRole/s3 id key) and the secret field (S3_SECRET_FIELD) must be provided together. A lone value is treated as a misconfiguration rather than silently ignored.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/provider/JCloudBlobStoreProvider.java:447

            throw new IllegalArgumentException(
                    "ManagedLedgerOffloadMaxBlockSizeInBytes cannot be less than 5MB for "
                            + config.getDriver() + " offload");
        }
    };

    static final CredentialBuilder S3_CREDENTIAL_BUILDER = (TieredStorageConfiguration config) -> {
        if (config.getCredentials() != null) {
            return;
        }
        // Credentials provided in the tiered storage configuration take
        // precedence over the environment variables. Offload policies carry
        // credentials under the s3-prefixed keys for every S3-compatible
        // driver (see OffloadPoliciesImpl), so those are the keys accepted.
        String configId = config.getConfigProperty(S3_ID_FIELD);
        String configSecret = config.getConfigProperty(S3_SECRET_FIELD);
        if (StringUtils.isNotBlank(configId) || StringUtils.isNotBlank(configSecret)) {
            if (StringUtils.isBlank(configId) || StringUtils.isBlank(configSecret)) {
                throw new IllegalArgumentException(
                        "Both " + S3_ID_FIELD + " and " + S3_SECRET_FIELD
                                + " must be set when providing offload credentials in the configuration");
            }
            Credentials credentials = new Credentials(configId, configSecret);
            config.setProviderCredentials(() -> credentials);
            return;
        }
        String accountName = System.getenv().getOrDefault("ACCESS_KEY_ID", "");
        // For forward compatibility
        if (StringUtils.isEmpty(accountName.trim())) {
            accountName = System.getenv().getOrDefault("ALIYUN_OSS_ACCESS_KEY_ID", "");
        }
        if (StringUtils.isEmpty(accountName.trim())) {
            throw new IllegalArgumentException("Couldn't get the access key id.");
        }
        String accountKey = System.getenv().getOrDefault("ACCESS_KEY_SECRET", "");
        if (StringUtils.isEmpty(accountKey.trim())) {
            accountKey = System.getenv().getOrDefault("ALIYUN_OSS_ACCESS_KEY_SECRET", "");

View on GitHub (pinned to 820761864e)

Solutions

1. Set both the id and secret config properties (e.g. s3ManagedLedgerOffloadAccessKeyId and s3ManagedLedgerOffloadAccessKeySecret) to non-blank values together.
2. Check configProperties via logs/startup for which of the two keys is missing or blank.
3. If credentials should come from the environment or instance profile instead, remove BOTH config keys so the env-based credential path is used.
4. Fix templating/secrets manager so both values render.

Example fix

// before
s3ManagedLedgerOffloadAccessKeyId=AKIA...
# secret missing
// after
s3ManagedLedgerOffloadAccessKeyId=AKIA...
s3ManagedLedgerOffloadAccessKeySecret=wJalr...

String id = cfg.getConfigProperty(S3_ID_FIELD);
String secret = cfg.getConfigProperty(S3_SECRET_FIELD);
if ((id != null && !id.isBlank()) || (secret != null && !secret.isBlank())) {
    if (id == null || id.isBlank() || secret == null || secret.isBlank()) {
        throw new IllegalStateException("Provide both S3 access key id and secret, or neither");
    }
}

try {
    provider.validateConfig(cfg);
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("must be set when providing offload credentials")) {
        log.error("S3 offload credential pair incomplete: set both id and secret, or remove both", e);
    }
}

When it happens

Trigger: During S3 credential building: config.getConfigProperty(S3_ID_FIELD) is non-blank while S3_SECRET_FIELD is blank, or vice versa — e.g. only the access key was added to configProperties, only the secret key leaked into broker.conf, or one key was renamed/typoed.

Common situations: Rotating credentials and only updating one of the two keys; pasting a credential pair but forgetting the second line; secrets templating that renders one variable empty; copy-pasting only the secret into configuration expecting the id to come from env.

Related errors

• Timeout during mark-delete operation
• Timeout during clear backlog operation
• Timeout during skip messages operation
• Timeout during delete operation
• Timeout during close operation

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/73940a750edfe192. Report an issue: GitHub.