apache/hadoop · error · IOException

Exception while initializing metric credentials

Error message

Exception while initializing metric credentials

What it means

AbfsMetricsManager wraps the SharedKeyCredentials constructor in a try/catch and rethrows IllegalArgumentException as IOException("Exception while initializing metric credentials ", e). SharedKeyCredentials throws IllegalArgumentException when the key is not valid Base64, so this is almost always a malformed fs.azure.metrics.account.key. It fires only when both metrics account name and key are set and metrics collection is enabled.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsMetricsManager.java:153

        String metricAccountKey = abfsConfiguration.getMetricAccountKey();
        this.metricFormat = abfsConfiguration.getMetricFormat();
        if (isNotEmpty(metricAccountName) && isNotEmpty(
            metricAccountKey)) {
          int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
          if (dotIndex <= 0) {
            throw new InvalidUriException(
                metricAccountName + " - account name is not fully qualified.");
          }
          try {
            metricSharedkeyCredentials = new SharedKeyCredentials(
                metricAccountName.substring(0, dotIndex),
                metricAccountKey);
            hasSeparateMetricAccount = true;
            setMetricsUrl(metricAccountName.startsWith(HTTPS_SCHEME)
                ? metricAccountName : HTTPS_SCHEME + COLON
                + FORWARD_SLASH + FORWARD_SLASH + metricAccountName);
          } catch (IllegalArgumentException e) {
            throw new IOException(
                "Exception while initializing metric credentials ", e);
          }
        } else {
          setMetricsUrl(baseUrlString.substring(0, indexLastForwardSlash + 1));
        }
        // Once the metric URL is set, initialize the metrics
        abfsCounters.initializeMetrics(metricFormat, abfsConfiguration);
        // Metrics emitter scheduler
        this.metricsEmitScheduler
            = Executors.newSingleThreadScheduledExecutor();
        // run every 1 minute to check the metrics count
        this.metricsEmitScheduler.scheduleWithFixedDelay(
            () -> {
              if (abfsCounters.getAbfsBackoffMetrics()
                  .getMetricValue(TOTAL_NUMBER_OF_REQUESTS)
                  >= abfsConfiguration.getMetricsEmitThreshold()) {
                emitCollectedMetrics();
              }

View on GitHub (pinned to 2add963021)

Solutions

Paste the exact Base64 access key from the Azure portal (Storage account > Access keys) into fs.azure.metrics.account.key
Verify locally: Base64.getDecoder().decode(key) must succeed without exception
Check for surrounding whitespace/quotes/newlines introduced by config files or secret managers
If a separate metrics account is not needed, unset both metrics account name and key

Example fix

# before (shell-injected, whitespace + wrong secret type)
fs.azure.metrics.account.key=" AccountKey='sv=2020-...'"   # SAS token, not a key

# after
fs.azure.metrics.account.key=<exact base64 key, no quotes or spaces>

Defensive patterns

Strategy: validation

Validate before calling

// Prove the key is valid Base64 before the FS tries to use it
String key = conf.get("fs.azure.metrics.account.key");
if (key != null) {
  try {
    java.util.Base64.getDecoder().decode(key.trim());
  } catch (IllegalArgumentException e) {
    throw new IllegalArgumentException(
        "fs.azure.metrics.account.key is not valid Base64", e);
  }
}

Try / catch

try {
  FileSystem fs = path.getFileSystem(conf);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith(
      "Exception while initializing metric credentials")) {
    // key is malformed: fix the secret, then re-create the FileSystem
  } else throw e;
}

Prevention

Copy access keys exactly from the Azure portal; never a connection string or SAS
Trim whitespace/newlines when injecting keys from secret managers
Test-decode the key in CI config validation

When it happens

Trigger: fs.azure.metrics.account.key containing a non-Base64 string: wrong key, truncated copy/paste, wrapped in quotes/whitespace, a SAS token pasted instead of the account key, or a key from a different (new-style) storage account whose key format does not parse.

Common situations: Secrets mangled by config management (extra newline, XML entity escaping, template placeholders); rotating keys and pasting the connection string instead of the key; CI configs injecting the wrong secret variable.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/83b4a261e1fb4eaf. Report an issue: GitHub.