{"record":{"id":"327db895aa208de3","repo":"apache/hadoop","slug":"error-logging-in-securely-ex","errorCode":null,"errorMessage":"Error logging in securely: [${ex}]","messagePattern":"Error logging in securely: \\[(.+?)\\]","errorType":"exception","errorClass":"MetricsException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java","lineNumber":261,"sourceCode":"    rollIntervalMillis = getRollInterval();\n\n    conf = loadConf();\n    UserGroupInformation.setConfiguration(conf);\n\n    // Don't do secure setup if it's not needed.\n    if (UserGroupInformation.isSecurityEnabled()) {\n      // Validate config so that we don't get an NPE\n      checkIfPropertyExists(KEYTAB_PROPERTY_KEY);\n      checkIfPropertyExists(USERNAME_PROPERTY_KEY);\n\n\n      try {\n        // Login as whoever we're supposed to be and let the hostname be pulled\n        // from localhost. If security isn't enabled, this does nothing.\n        SecurityUtil.login(conf, properties.getString(KEYTAB_PROPERTY_KEY),\n            properties.getString(USERNAME_PROPERTY_KEY));\n      } catch (IOException ex) {\n        throw new MetricsException(\"Error logging in securely: [\"\n            + ex.toString() + \"]\", ex);\n      }\n    }\n  }\n\n  /**\n   * Initialize the connection to HDFS and create the base directory. Also\n   * launch the flush thread.\n   */\n  private boolean initFs() {\n    boolean success = false;\n\n    fileSystem = getFileSystem();\n\n    // This step isn't strictly necessary, but it makes debugging issues much\n    // easier. We try to create the base directory eagerly and fail with\n    // copious debug info if it fails.\n    try {","sourceCodeStart":243,"sourceCodeEnd":279,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java#L243-L279","documentation":"When the JVM runs with Kerberos security enabled (UserGroupInformation.isSecurityEnabled()), RollingFileSystemSink.init() calls SecurityUtil.login(conf, <value of keytab-key>, <value of principal-key>). Those two sink properties hold the NAMES of Configuration keys (e.g. dfs.namenode.keytab.file / dfs.namenode.kerberos.principal) that resolve the actual keytab path and principal. An IOException from the login — missing or unreadable keytab, principal mismatch, bad _HOST interpolation, KDC unreachable — is wrapped as MetricsException(\"Error logging in securely: [...]\") and sink init fails.","triggerScenarios":"keytab-key/principal-key are set and present, but the Configuration keys they reference point to a keytab file that does not exist on the node or is unreadable by the daemon user; the principal does not match the keytab (wrong realm, hostname change breaking _HOST); krb5.conf or KDC misconfigured.","commonSituations":"Kerberizing a cluster and wiring the metrics sink to daemon keytab properties that are not in the effective Configuration; keytab not deployed to every node; realm rename after a merge; hostname not resolving to the FQDN used in the principal.","solutions":["Validate the pair as the daemon user: kinit -kt /etc/security/keytabs/nn.keytab nn/_HOST@REALM — if kinit fails, fix the keytab/principal first","Ensure the values of keytab-key and principal-key exactly name existing Configuration keys whose values are the real keytab path and principal","Check the keytab file is readable by the daemon user (ls -l, stat) and present on this node","Verify krb5.conf realm/KDC settings and that _HOST resolves to the host's FQDN"],"exampleFix":"# before\nnamenode.sink.rolling.class=org.apache.hadoop.metrics2.sink.RollingFileSystemSink\nnamenode.sink.rolling.keytab-key=dfs.namenode.keytab.file   # key absent from hdfs-site.xml\n\n# after\n# add to hdfs-site.xml: dfs.namenode.keytab.file=/etc/security/keytabs/nn.service.keytab\nnamenode.sink.rolling.class=org.apache.hadoop.metrics2.sink.RollingFileSystemSink\nnamenode.sink.rolling.keytab-key=dfs.namenode.keytab.file\nnamenode.sink.rolling.principal-key=dfs.namenode.kerberos.principal","handlingStrategy":"validation","validationCode":"// verify the keytab/principal pair the sink will use, as the daemon user\nString keytab = conf.get(conf.get(\"namenode.sink.rolling.keytab-key\"));\nString principal = conf.get(conf.get(\"namenode.sink.rolling.principal-key\"));\nif (keytab == null || principal == null || !Files.isReadable(Paths.get(keytab))) {\n  throw new IllegalStateException(\"RollingFileSystemSink login prereqs missing: keytab=\"\n      + keytab + \" principal=\" + principal);\n}","typeGuard":null,"tryCatchPattern":"try {\n  sink.init(subsetConf);\n} catch (MetricsException e) {\n  // message embeds the SecurityUtil.login IOException (missing keytab, bad principal, KDC down)\n  LOG.error(\"Secure login for metrics sink failed: {}\", e.getMessage(), e.getCause());\n}","preventionTips":["Test the exact keytab/principal pair with kinit -kt <keytab> <principal> as the daemon user before enabling the sink","Keep keytab-key/principal-key values pointed at the same Configuration keys the daemon itself logs in with","Deploy keytabs to every node and keep file permissions readable only by the daemon user"],"tags":["kerberos","security","metrics2","rolling-file-sink","authentication"],"backgroundTag":"kerberos-authentication-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}