apache/hadoop · error · MetricsException

Metrics2 configuration is missing ${key} property

Error message

Metrics2 configuration is missing ${key} property

What it means

With Kerberos security enabled, RollingFileSystemSink.init() requires two extra sink properties: keytab-key and principal-key, whose VALUES are names of Configuration keys that hold the actual keytab path and principal (e.g. yarn.nodemanager.keytab / yarn.nodemanager.principal). checkIfPropertyExists throws MetricsException("Metrics2 configuration is missing <key> property") naming exactly which of the two is absent from the sink's configuration subset.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java:426

  private long getNonNegative(String key, int defaultValue) {
    int flushOffsetIntervalMillis = properties.getInt(key, defaultValue);

    if (flushOffsetIntervalMillis < 0) {
      throw new MetricsException("The " + key + " property must be "
          + "non-negative. Value was " + flushOffsetIntervalMillis);
    }

    return flushOffsetIntervalMillis;
  }

  /**
   * Throw a {@link MetricsException} if the given property is not set.
   *
   * @param key the key to validate
   */
  private void checkIfPropertyExists(String key) {
    if (!properties.containsKey(key)) {
      throw new MetricsException("Metrics2 configuration is missing " + key
          + " property");
    }
  }

  /**
   * Return the supplied configuration for testing or otherwise load a new
   * configuration.
   *
   * @return the configuration to use
   */
  private Configuration loadConf() {
    Configuration c;

    if (suppliedConf != null) {
      c = suppliedConf;
    } else {
      // The config we're handed in init() isn't the one we want here, so we
      // create a new one to pick up the full settings.

View on GitHub (pinned to 2add963021)

Solutions

1. Add both properties, pointing at existing Configuration keys: <prefix>.sink.<instance>.keytab-key=dfs.namenode.keytab.file and .principal-key=dfs.namenode.kerberos.principal
2. Verify the referenced keys exist in the effective (merged, XML) Configuration and are not filtered out
3. Confirm the property prefix matches the sink instance name exactly

Example fix

# before (secure cluster)
namenode.sink.rolling.class=org.apache.hadoop.metrics2.sink.RollingFileSystemSink
# -> Metrics2 configuration is missing keytab-key property

# after
namenode.sink.rolling.class=org.apache.hadoop.metrics2.sink.RollingFileSystemSink
namenode.sink.rolling.keytab-key=dfs.namenode.keytab.file
namenode.sink.rolling.principal-key=dfs.namenode.kerberos.principal

if (UserGroupInformation.isSecurityEnabled()) {
  for (String key : new String[] {"keytab-key", "principal-key"}) {
    if (!subsetConf.containsKey(key)) {
      throw new IllegalStateException("Secure cluster: sink config missing " + key
          + " (value must name a Configuration key holding the "
          + (key.equals("keytab-key") ? "keytab path" : "principal") + ")");
    }
  }
}

try {
  sink.init(subsetConf);
} catch (MetricsException e) {
  // 'Metrics2 configuration is missing <key> property' — add keytab-key and/or principal-key
  LOG.error("Add the missing secure-login sink property: {}", e.getMessage());
}

When it happens

Trigger: Kerberized cluster with the rolling sink configured, but <prefix>.sink.<instance>.keytab-key or .principal-key (or both) omitted from hadoop-metrics2.properties.

Common situations: Enabling security on an existing cluster and adding the rolling sink with only basepath/roll-interval; renaming the sink instance so old property prefixes no longer match (SubsetConfiguration keys are prefix-dependent).

Related errors

• Error logging in securely: [${ex}]
• Secure IO is not possible without native code extensions.
• Server asks us to fall back to SIMPLE auth, but this client
• Failed to create ${basePath}[source=${source}, allow-append=
• Unrecognized flush interval: ${rollInterval}. Must be a numb

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1e7404c742b8702c. Report an issue: GitHub.