apache/iceberg · error · RuntimeMetaException

Failed to reconnect to Hive Metastore

Error message

Failed to reconnect to Hive Metastore

What it means

HiveClientPool.reconnect() wraps MetaException from IMetaStoreClient.reconnect() in RuntimeMetaException with message 'Failed to reconnect to Hive Metastore'. It fires when an existing pooled client loses its session and the client-side reconnect also fails, meaning the metastore is unreachable or the session cannot be re-established with current credentials.

Source

Thrown at hive-metastore/src/main/java/org/apache/iceberg/hive/HiveClientPool.java:95

          && t.getMessage().contains("Another instance of Derby may have already booted")) {
        throw new RuntimeMetaException(
            t,
            "Failed to start an embedded metastore because embedded "
                + "Derby supports only one client at a time. To fix this, use a metastore that supports "
                + "multiple clients.");
      }

      throw new RuntimeMetaException(t, "Failed to connect to Hive Metastore");
    }
  }

  @Override
  protected IMetaStoreClient reconnect(IMetaStoreClient client) {
    try {
      client.close();
      client.reconnect();
    } catch (MetaException e) {
      throw new RuntimeMetaException(e, "Failed to reconnect to Hive Metastore");
    }
    return client;
  }

  @Override
  protected boolean isConnectionException(Exception e) {
    return super.isConnectionException(e)
        || (e instanceof MetaException
            && e.getMessage()
                .contains("Got exception: org.apache.thrift.transport.TTransportException"));
  }

  @Override
  protected void close(IMetaStoreClient client) {
    client.close();
  }

  @VisibleForTesting

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check metastore service health/logs and restart or restore it if it went down.
2. Renew Kerberos credentials (kinit / refreshed keytab) so reconnect can re-authenticate.
3. Retry the operation after the pool replaces the failed client — the pool's failure-retry logic may succeed once connectivity returns.
4. Verify network stability (timeouts, idle connection limits, load balancer idle timeouts) between client and metastore.

Example fix

// before
Table table = catalog.loadTable(identifier); // fails mid-run after metastore restart
// after
Table table;
try {
  table = catalog.loadTable(identifier);
} catch (RuntimeMetaException e) {
  // renew credentials / wait for metastore, then retry
  table = Tasks.retry(3).run(() -> catalog.loadTable(identifier));
}

// periodic metastore health probe during long jobs
boolean healthy = false;
try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(host, port), 3000);
  healthy = true;
}

try {
  return catalog.loadTable(identifier);
} catch (RuntimeMetaException e) {
  if (e.getMessage().contains("reconnect")) {
    // renew Kerberos creds / wait for metastore recovery, retry with backoff
    return Tasks.retry(3).exponentialBackoff(1000, 60000).run(() -> catalog.loadTable(identifier));
  }
  throw e;
}

When it happens

Trigger: A long-lived pooled HiveMetaStoreClient hits a connection failure mid-operation, the pool calls reconnect(client), and client.reconnect() throws MetaException — metastore restarted/degraded, network blip, or expired Kerberos credentials preventing re-authentication.

Common situations: Metastore restart or failover while a job is running; network partitions between compute and metastore; long-running Spark jobs whose Kerberos tickets expire mid-execution; metastore under load dropping connections.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• Failed to list all tables under namespace ${namespace}
• Failed to connect to Hive Metastore
• The view %s.%s has been modified concurrently
• Failed to acquire locks from metastore because the underlyin
• Metastore operation failed for %s.%s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1ebc82fa67c79652. Report an issue: GitHub.