apache/flink · error · KeyProviderException

Unable to retrieve Azure storage key from environment. "{}"

Error message

Unable to retrieve Azure storage key from environment. "{}" not set.

What it means

Thrown by EnvironmentVariableKeyProvider.getStorageAccountKey when the AZURE_STORAGE_KEY environment variable is not set in the JVM's environment. This provider is a simple strategy for giving the Hadoop ABFS driver the storage account key; if the variable is absent it raises KeyProviderException during file system initialization.

Source

Thrown at flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/EnvironmentVariableKeyProvider.java:42

/**
 * An implementation of {@link org.apache.hadoop.fs.azure.KeyProvider}, which reads the Azure
 * storage key from an environment variable named "AZURE_STORAGE_KEY".
 */
public class EnvironmentVariableKeyProvider implements KeyProvider {

    public static final String AZURE_STORAGE_KEY_ENV_VARIABLE = "AZURE_STORAGE_KEY";

    @Override
    public String getStorageAccountKey(final String s, final Configuration configuration)
            throws KeyProviderException {

        String azureStorageKey = System.getenv(AZURE_STORAGE_KEY_ENV_VARIABLE);

        if (azureStorageKey != null) {
            return azureStorageKey;
        } else {
            throw new KeyProviderException(
                    "Unable to retrieve Azure storage key from environment. \""
                            + AZURE_STORAGE_KEY_ENV_VARIABLE
                            + "\" not set.");
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Export AZURE_STORAGE_KEY in the environment of EVERY Flink process (JobManager, TaskManager, CLI clients that touch the FS)
2. In Kubernetes, add it via env in the JM/TM pod specs or as a secret reference; for scripts, use env in flink-conf 'env.java.opts.all' only if unavoidable — prefer real process env
3. If you cannot control the environment, switch to SimpleKeyProvider-style configuration where the key lives in the Hadoop config (fs.azure.account.key.<account>) or use a custom KeyProvider reading a secret store
4. Verify with a trivial check: 'flink run' a job that calls System.getenv("AZURE_STORAGE_KEY")

Example fix

# before (kubernetes pod spec has no env)
containers:
  - name: flink-taskmanager

# after
containers:
  - name: flink-taskmanager
    env:
      - name: AZURE_STORAGE_KEY
        valueFrom:
          secretKeyRef:
            name: azure-storage
            key: storage-key

// before starting Flink processes
String key = System.getenv("AZURE_STORAGE_KEY");
if (key == null || key.isEmpty()) {
    throw new IllegalStateException(
        "AZURE_STORAGE_KEY must be set for EnvironmentVariableKeyProvider");
}

try {
    keyProvider.getStorageAccountKey(account, conf);
} catch (org.apache.flink.fs.azurefs.KeyProviderException e) {
    // env var missing on this process: fix the process environment, not the code
    failFastWithEnvInstructions();
}

When it happens

Trigger: Configuring flink-azure-fs-hadoop with the environment-variable key provider (fs.azure.account.keyprovider.<account> = org.apache.flink.fs.azurefs.EnvironmentVariableKeyProvider) and then running the JobManager/TaskManager process without AZURE_STORAGE_KEY exported.

Common situations: Works in a shell but fails under systemd/Kubernetes/Docker where the variable was never injected into the service environment; variable set only on one node type; typo in the variable name.

Related errors

• The configuration directory '{}', specified in the '{}' envi
• The configuration directory was not specified. Please specif
• The given configuration directory name '{}' ({}) does not de
• fs.s3.aws.credentials.provider is set but contains no valid
• Class {} does not implement AwsCredentialsProvider

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/b234cff2a2ad30c9. Report an issue: GitHub.