apache/hadoop · error · IOException

Exception while initializing metric credentials

Error message

Exception while initializing metric credentials

What it means

AbfsMetricsManager wraps the SharedKeyCredentials constructor in a try/catch and rethrows IllegalArgumentException as IOException("Exception while initializing metric credentials ", e). SharedKeyCredentials throws IllegalArgumentException when the key is not valid Base64, so this is almost always a malformed fs.azure.metrics.account.key. It fires only when both metrics account name and key are set and metrics collection is enabled.

Solutions

Paste the exact Base64 access key from the Azure portal (Storage account > Access keys) into fs.azure.metrics.account.key
Verify locally: Base64.getDecoder().decode(key) must succeed without exception
Check for surrounding whitespace/quotes/newlines introduced by config files or secret managers
If a separate metrics account is not needed, unset both metrics account name and key

Example fix

# before (shell-injected, whitespace + wrong secret type)
fs.azure.metrics.account.key=" AccountKey='sv=2020-...'"   # SAS token, not a key

# after
fs.azure.metrics.account.key=<exact base64 key, no quotes or spaces>

Defensive patterns

Strategy: validation

Validate before calling

// Prove the key is valid Base64 before the FS tries to use it
String key = conf.get("fs.azure.metrics.account.key");
if (key != null) {
  try {
    java.util.Base64.getDecoder().decode(key.trim());
  } catch (IllegalArgumentException e) {
    throw new IllegalArgumentException(
        "fs.azure.metrics.account.key is not valid Base64", e);
  }
}

Try / catch

try {
  FileSystem fs = path.getFileSystem(conf);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith(
      "Exception while initializing metric credentials")) {
    // key is malformed: fix the secret, then re-create the FileSystem
  } else throw e;
}

Prevention

Copy access keys exactly from the Azure portal; never a connection string or SAS
Trim whitespace/newlines when injecting keys from secret managers
Test-decode the key in CI config validation

When it happens

Trigger: fs.azure.metrics.account.key containing a non-Base64 string: wrong key, truncated copy/paste, wrapped in quotes/whitespace, a SAS token pasted instead of the account key, or a key from a different (new-style) storage account whose key format does not parse.

Common situations: Secrets mangled by config management (extra newline, XML entity escaping, template placeholders); rotating keys and pasting the connection string instead of the key; CI configs injecting the wrong secret variable.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/83b4a261e1fb4eaf. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsMetricsManager.java:153

        String metricAccountKey = abfsConfiguration.getMetricAccountKey();
        this.metricFormat = abfsConfiguration.getMetricFormat();
        if (isNotEmpty(metricAccountName) && isNotEmpty(
            metricAccountKey)) {
          int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
          if (dotIndex <= 0) {
            throw new InvalidUriException(
                metricAccountName + " - account name is not fully qualified.");
          }
          try {
            metricSharedkeyCredentials = new SharedKeyCredentials(
                metricAccountName.substring(0, dotIndex),
                metricAccountKey);
            hasSeparateMetricAccount = true;
            setMetricsUrl(metricAccountName.startsWith(HTTPS_SCHEME)
                ? metricAccountName : HTTPS_SCHEME + COLON
                + FORWARD_SLASH + FORWARD_SLASH + metricAccountName);
          } catch (IllegalArgumentException e) {
            throw new IOException(
                "Exception while initializing metric credentials ", e);
          }
        } else {
          setMetricsUrl(baseUrlString.substring(0, indexLastForwardSlash + 1));
        }
        // Once the metric URL is set, initialize the metrics
        abfsCounters.initializeMetrics(metricFormat, abfsConfiguration);
        // Metrics emitter scheduler
        this.metricsEmitScheduler
            = Executors.newSingleThreadScheduledExecutor();
        // run every 1 minute to check the metrics count
        this.metricsEmitScheduler.scheduleWithFixedDelay(
            () -> {
              if (abfsCounters.getAbfsBackoffMetrics()
                  .getMetricValue(TOTAL_NUMBER_OF_REQUESTS)
                  >= abfsConfiguration.getMetricsEmitThreshold()) {
                emitCollectedMetrics();
              }

View on GitHub (pinned to 2add963021)