{"record":{"id":"b80ed7f74e5d62e6","repo":"apache/seatunnel","slug":"kerberos-re-login-for-hivemetastore-failed","errorCode":null,"errorMessage":"Kerberos re-login for HiveMetaStore failed: {}","messagePattern":"Kerberos re-login for HiveMetaStore failed: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-connectors-v2/connector-hive/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreCatalog.java","lineNumber":707,"sourceCode":"    }\n\n    @Override\n    public synchronized void close() throws CatalogException {\n        if (Objects.nonNull(hiveClient)) {\n            hiveClient.close();\n        }\n    }\n\n    private void maybeRelogin() {\n        if (userGroupInformation == null) {\n            return;\n        }\n        try {\n            if (userGroupInformation.isFromKeytab()) {\n                userGroupInformation.checkTGTAndReloginFromKeytab();\n            }\n        } catch (Exception e) {\n            log.warn(\"Kerberos re-login for HiveMetaStore failed: {}\", e.getMessage());\n        }\n    }\n\n    private CatalogTable convertHiveTableToCatalogTable(Table hiveTable) {\n        List<org.apache.seatunnel.api.table.catalog.Column> columns = new ArrayList<>();\n\n        if (hiveTable.getSd() != null && hiveTable.getSd().getCols() != null) {\n            for (org.apache.hadoop.hive.metastore.api.FieldSchema field :\n                    hiveTable.getSd().getCols()) {\n                org.apache.seatunnel.api.table.type.SeaTunnelDataType<?> dataType =\n                        HiveTypeConvertor.covertHiveTypeToSeaTunnelType(\n                                field.getName(), field.getType());\n                columns.add(\n                        org.apache.seatunnel.api.table.catalog.PhysicalColumn.of(\n                                field.getName(), dataType, 0, true, null, field.getComment()));\n            }\n        }\n","sourceCodeStart":689,"sourceCodeEnd":725,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-hive/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreCatalog.java#L689-L725","documentation":"HiveMetaStoreCatalog.maybeRelogin, invoked from getClient, attempts a Kerberos re-login via UserGroupInformation.checkTGTAndReloginFromKeytab() when the UGI was created from a keytab. If the re-login attempt itself throws (I/O error reading the keytab, KDC unreachable, principal problems), the code logs this warning and swallows the exception, proceeding with the possibly-stale UGI. The next HMS RPC may then fail with a Kerberos/GSS authentication error.","triggerScenarios":"A long-running job whose Kerberos ticket lifetime (max lifetime / renew lifetime in KDC) expires while isFromKeytab() is true and checkTGTAndReloginFromKeytab() fails to refresh — e.g. keytab file moved or permission changed, KDC temporarily unavailable, clock skew between nodes exceeding allowed tolerance.","commonSituations":"Jobs running longer than the Kerberos ticket lifetime (default often 24h); keytab not distributed to all worker nodes or path wrong in krb5.conf/hive config; clock drift causing 'Clock skew too great' KDC errors; Hadoop security (hadoop.security.authentication=kerberos) misconfigured.","solutions":["Verify the keytab exists and is readable by the job user on every node: check the principal and keytab path passed to UserGroupInformation.loginUserFromKeytab.","Synchronize clocks across cluster nodes with NTP/chrony to avoid KDC clock-skew rejections.","Confirm krb5.conf is correct and the KDC is reachable from worker nodes; test with `kinit -kt /path/keytab principal`.","Check KDC ticket lifetimes (max_renewable_life) allow re-login; if tickets cannot renew, reduce job duration or renew tickets externally.","Watch for subsequent 'Failed to specify server's Kerberos principal' or GSSException errors in getClient calls — those are the downstream symptoms of this swallowed warning."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Pre-flight Kerberos check before job start:\nUserGroupInformation ugi = UserGroupInformation.getLoginUser();\nif (ugi.isFromKeytab()) {\n    Process k = new ProcessBuilder(\"klist\", \"-kt\", keytabPath).start();\n    if (k.waitFor() != 0) throw new IllegalStateException(\"Keytab unreadable: \" + keytabPath);\n}","typeGuard":null,"tryCatchPattern":"// maybeRelogin swallows the exception, so guard the downstream HMS call instead:\ntry {\n    IMetaStoreClient client = catalog.getClient(); // triggers maybeRelogin\n    client.getAllDatabases();\n} catch (TException | IOException e) {\n    // Likely stale Kerberos ticket after failed re-login\n    UserGroupInformation.getLoginUser().reloginFromKeytab();\n    throw new CatalogException(\"HMS auth failed after Kerberos re-login problem\", e);\n}","preventionTips":["Distribute the keytab to every worker node with correct file permissions for the job user.","Run NTP/chrony on all nodes to prevent KDC clock-skew rejections.","Set hadoop.kerberos.min.seconds.before.relogin below the ticket lifetime so re-login happens proactively.","Confirm KDC max_renewable_life supports ticket renewal for the job duration."],"tags":["kerberos","authentication","hive","keytab","security"],"backgroundTag":"jwt-token-expired","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}