{"record":{"id":"1ebc82fa67c79652","repo":"apache/iceberg","slug":"failed-to-reconnect-to-hive-metastore","errorCode":null,"errorMessage":"Failed to reconnect to Hive Metastore","messagePattern":"Failed to reconnect to Hive Metastore","errorType":"exception","errorClass":"RuntimeMetaException","httpStatus":null,"severity":"error","filePath":"hive-metastore/src/main/java/org/apache/iceberg/hive/HiveClientPool.java","lineNumber":95,"sourceCode":"          && t.getMessage().contains(\"Another instance of Derby may have already booted\")) {\n        throw new RuntimeMetaException(\n            t,\n            \"Failed to start an embedded metastore because embedded \"\n                + \"Derby supports only one client at a time. To fix this, use a metastore that supports \"\n                + \"multiple clients.\");\n      }\n\n      throw new RuntimeMetaException(t, \"Failed to connect to Hive Metastore\");\n    }\n  }\n\n  @Override\n  protected IMetaStoreClient reconnect(IMetaStoreClient client) {\n    try {\n      client.close();\n      client.reconnect();\n    } catch (MetaException e) {\n      throw new RuntimeMetaException(e, \"Failed to reconnect to Hive Metastore\");\n    }\n    return client;\n  }\n\n  @Override\n  protected boolean isConnectionException(Exception e) {\n    return super.isConnectionException(e)\n        || (e instanceof MetaException\n            && e.getMessage()\n                .contains(\"Got exception: org.apache.thrift.transport.TTransportException\"));\n  }\n\n  @Override\n  protected void close(IMetaStoreClient client) {\n    client.close();\n  }\n\n  @VisibleForTesting","sourceCodeStart":77,"sourceCodeEnd":113,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveClientPool.java#L77-L113","documentation":"HiveClientPool.reconnect() wraps MetaException from IMetaStoreClient.reconnect() in RuntimeMetaException with message 'Failed to reconnect to Hive Metastore'. It fires when an existing pooled client loses its session and the client-side reconnect also fails, meaning the metastore is unreachable or the session cannot be re-established with current credentials.","triggerScenarios":"A long-lived pooled HiveMetaStoreClient hits a connection failure mid-operation, the pool calls reconnect(client), and client.reconnect() throws MetaException — metastore restarted/degraded, network blip, or expired Kerberos credentials preventing re-authentication.","commonSituations":"Metastore restart or failover while a job is running; network partitions between compute and metastore; long-running Spark jobs whose Kerberos tickets expire mid-execution; metastore under load dropping connections.","solutions":["Check metastore service health/logs and restart or restore it if it went down.","Renew Kerberos credentials (kinit / refreshed keytab) so reconnect can re-authenticate.","Retry the operation after the pool replaces the failed client — the pool's failure-retry logic may succeed once connectivity returns.","Verify network stability (timeouts, idle connection limits, load balancer idle timeouts) between client and metastore."],"exampleFix":"// before\nTable table = catalog.loadTable(identifier); // fails mid-run after metastore restart\n// after\nTable table;\ntry {\n  table = catalog.loadTable(identifier);\n} catch (RuntimeMetaException e) {\n  // renew credentials / wait for metastore, then retry\n  table = Tasks.retry(3).run(() -> catalog.loadTable(identifier));\n}","handlingStrategy":"retry","validationCode":"// periodic metastore health probe during long jobs\nboolean healthy = false;\ntry (Socket s = new Socket()) {\n  s.connect(new InetSocketAddress(host, port), 3000);\n  healthy = true;\n}","typeGuard":null,"tryCatchPattern":"try {\n  return catalog.loadTable(identifier);\n} catch (RuntimeMetaException e) {\n  if (e.getMessage().contains(\"reconnect\")) {\n    // renew Kerberos creds / wait for metastore recovery, retry with backoff\n    return Tasks.retry(3).exponentialBackoff(1000, 60000).run(() -> catalog.loadTable(identifier));\n  }\n  throw e;\n}","preventionTips":["Configure Kerberos credential renewal for long-running jobs","Add metastore health monitoring and alerting","Use retry-with-backoff around catalog operations that can survive transient metastore restarts"],"tags":["hive","metastore","reconnect","network","kerberos"],"backgroundTag":"connection-refused","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}