apache/hadoop · error · MetricsException

Error logging in securely: [${ex}]

Error message

Error logging in securely: [${ex}]

What it means

When the JVM runs with Kerberos security enabled (UserGroupInformation.isSecurityEnabled()), RollingFileSystemSink.init() calls SecurityUtil.login(conf, <value of keytab-key>, <value of principal-key>). Those two sink properties hold the NAMES of Configuration keys (e.g. dfs.namenode.keytab.file / dfs.namenode.kerberos.principal) that resolve the actual keytab path and principal. An IOException from the login — missing or unreadable keytab, principal mismatch, bad _HOST interpolation, KDC unreachable — is wrapped as MetricsException("Error logging in securely: [...]") and sink init fails.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java:261

    rollIntervalMillis = getRollInterval();

    conf = loadConf();
    UserGroupInformation.setConfiguration(conf);

    // Don't do secure setup if it's not needed.
    if (UserGroupInformation.isSecurityEnabled()) {
      // Validate config so that we don't get an NPE
      checkIfPropertyExists(KEYTAB_PROPERTY_KEY);
      checkIfPropertyExists(USERNAME_PROPERTY_KEY);


      try {
        // Login as whoever we're supposed to be and let the hostname be pulled
        // from localhost. If security isn't enabled, this does nothing.
        SecurityUtil.login(conf, properties.getString(KEYTAB_PROPERTY_KEY),
            properties.getString(USERNAME_PROPERTY_KEY));
      } catch (IOException ex) {
        throw new MetricsException("Error logging in securely: ["
            + ex.toString() + "]", ex);
      }
    }
  }

  /**
   * Initialize the connection to HDFS and create the base directory. Also
   * launch the flush thread.
   */
  private boolean initFs() {
    boolean success = false;

    fileSystem = getFileSystem();

    // This step isn't strictly necessary, but it makes debugging issues much
    // easier. We try to create the base directory eagerly and fail with
    // copious debug info if it fails.
    try {

View on GitHub (pinned to 2add963021)

Solutions

1. Validate the pair as the daemon user: kinit -kt /etc/security/keytabs/nn.keytab nn/_HOST@REALM — if kinit fails, fix the keytab/principal first
2. Ensure the values of keytab-key and principal-key exactly name existing Configuration keys whose values are the real keytab path and principal
3. Check the keytab file is readable by the daemon user (ls -l, stat) and present on this node
4. Verify krb5.conf realm/KDC settings and that _HOST resolves to the host's FQDN

Example fix

# before
namenode.sink.rolling.class=org.apache.hadoop.metrics2.sink.RollingFileSystemSink
namenode.sink.rolling.keytab-key=dfs.namenode.keytab.file   # key absent from hdfs-site.xml

# after
# add to hdfs-site.xml: dfs.namenode.keytab.file=/etc/security/keytabs/nn.service.keytab
namenode.sink.rolling.class=org.apache.hadoop.metrics2.sink.RollingFileSystemSink
namenode.sink.rolling.keytab-key=dfs.namenode.keytab.file
namenode.sink.rolling.principal-key=dfs.namenode.kerberos.principal

// verify the keytab/principal pair the sink will use, as the daemon user
String keytab = conf.get(conf.get("namenode.sink.rolling.keytab-key"));
String principal = conf.get(conf.get("namenode.sink.rolling.principal-key"));
if (keytab == null || principal == null || !Files.isReadable(Paths.get(keytab))) {
  throw new IllegalStateException("RollingFileSystemSink login prereqs missing: keytab="
      + keytab + " principal=" + principal);
}

try {
  sink.init(subsetConf);
} catch (MetricsException e) {
  // message embeds the SecurityUtil.login IOException (missing keytab, bad principal, KDC down)
  LOG.error("Secure login for metrics sink failed: {}", e.getMessage(), e.getCause());
}

When it happens

Trigger: keytab-key/principal-key are set and present, but the Configuration keys they reference point to a keytab file that does not exist on the node or is unreadable by the daemon user; the principal does not match the keytab (wrong realm, hostname change breaking _HOST); krb5.conf or KDC misconfigured.

Common situations: Kerberizing a cluster and wiring the metrics sink to daemon keytab properties that are not in the effective Configuration; keytab not deployed to every node; realm rename after a merge; hostname not resolving to the FQDN used in the principal.

Related errors

• Metrics2 configuration is missing ${key} property
• Server asks us to fall back to SIMPLE auth, but this client
• Security enabled but user not authenticated by filter
• Failed to obtain user group information: {}
• The client is configured to only allow connecting to secure

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/327db895aa208de3. Report an issue: GitHub.