apache/pulsar · error · RuntimeException

Failed to open state table for function ${tenant}/${namespac

Error message

Failed to open state table for function ${tenant}/${namespace}/${name} within timeout period

What it means

BKStateStoreProviderImpl.openStateTable throws RuntimeException on TimeoutException while opening the function's state table via the table service within the retry/timeout budget (retrying every 100 ms on internal server errors). If the table still cannot be opened after the loop, it also throws an IOException with a similar message; this error specifically surfaces the timeout variant. The table service could not serve an open-able view of the table in time.

Source

Thrown at pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java:194

                .attr("name", name)
                .log("Opening state table for function");
        // NOTE: this is a workaround until we bump bk version to 4.9.0
        // table might just be created above, so it might not be ready for serving traffic
        Stopwatch openSw = Stopwatch.createStarted();
        while (openSw.elapsed(TimeUnit.MINUTES) < 1) {
            try {
                return result(client.openTable(name), 1, TimeUnit.MINUTES);
            } catch (InternalServerException ise) {
                log.warn()
                        .attr("tenant", tenant)
                        .attr("namespace", namespace)
                        .attr("name", name)
                        .attr("message", ise.getMessage())
                        .log("Encountered internal server on opening state table,"
                                + " re-attempt in 100 milliseconds");
                TimeUnit.MILLISECONDS.sleep(100);
            } catch (TimeoutException e) {
                throw new RuntimeException(
                        "Failed to open state table for function " + tenant + "/" + namespace + "/" + name
                                + " within timeout period", e);
            }
        }
        throw new IOException("Failed to open state table for function " + tenant + "/" + namespace + "/" + name);
    }

    @SuppressWarnings("unchecked")
    @Override
    public <T extends StateStore> T getStateStore(String tenant, String namespace, String name) throws Exception {
        // we defer creation of the state table until a java instance is running here.
        createStateTable(stateStorageServiceUrl, tenant, namespace, name);
        Table<ByteBuf, ByteBuf> table = openStateTable(tenant, namespace, name);
        return (T) new BKStateStoreImpl(tenant, namespace, name, table);
    }

    @Override
    public void cleanUp(String tenant, String namespace, String name) throws Exception {

View on GitHub (pinned to 820761864e)

Solutions

1. Inspect the TimeoutException cause and preceding internal-server-error logs from the table service.
2. Check BookKeeper storage cluster health (bookies up, proxy reachable) and wait/retry — table recovery may complete shortly.
3. Verify network/DNS connectivity from the function instance to the table service proxy.
4. Restart the function instance after the storage cluster stabilizes; if persistent, investigate the table's bookie ensemble.

Example fix

// before
Table<ByteBuf, ByteBuf> table = provider.getStateStore(...); // blocks, may time out once
// after
Table<ByteBuf, ByteBuf> table = retryWithBackoff(3, () -> provider.getStateStore(...)); // tolerate transient table-service recovery

// check the table service is responsive before opening
// probe cluster health via broker admin or storage proxy metrics endpoint
boolean storageHealthy = probeHttp(storageProxyUrl + "/metrics");

try {
    Table<ByteBuf, ByteBuf> t = openStateTable(tenant, ns, name);
} catch (RuntimeException e) {
    if (e.getCause() instanceof TimeoutException || hasInternalServerErrors(e)) {
        // table service recovering: wait and retry once before failing the instance
        sleepSeconds(5);
        return openStateTable(tenant, ns, name);
    }
    throw e;
}

When it happens

Trigger: openStateTable (called via table(), used by getStateStore) repeatedly hits InternalServerError (retried with 100 ms sleeps) until a TimeoutException terminates the loop, or the underlying open times out directly.

Common situations: BookKeeper table service (proxy/storage) overloaded or restarting; table recovering/rebalancing during broker failover; state table owned by an unavailable bookie; DNS/network issues from the function instance to the storage proxy.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to setup / verify state table for function %s/%s/%s w
• Timeout during managed ledger close
• Timeout during managed ledger delete operation
• Failed to increment key '%s' by amount '%s'
• Failed to retrieve counter from key '%s'

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/69e0db98ac4708c7. Report an issue: GitHub.