{"record":{"id":"69e0db98ac4708c7","repo":"apache/pulsar","slug":"failed-to-open-state-table-for-function-tenant","errorCode":null,"errorMessage":"Failed to open state table for function ${tenant}/${namespace}/${name} within timeout period","messagePattern":"Failed to open state table for function (.+?)/(.+?)/(.+?) within timeout period","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java","lineNumber":194,"sourceCode":"                .attr(\"name\", name)\n                .log(\"Opening state table for function\");\n        // NOTE: this is a workaround until we bump bk version to 4.9.0\n        // table might just be created above, so it might not be ready for serving traffic\n        Stopwatch openSw = Stopwatch.createStarted();\n        while (openSw.elapsed(TimeUnit.MINUTES) < 1) {\n            try {\n                return result(client.openTable(name), 1, TimeUnit.MINUTES);\n            } catch (InternalServerException ise) {\n                log.warn()\n                        .attr(\"tenant\", tenant)\n                        .attr(\"namespace\", namespace)\n                        .attr(\"name\", name)\n                        .attr(\"message\", ise.getMessage())\n                        .log(\"Encountered internal server on opening state table,\"\n                                + \" re-attempt in 100 milliseconds\");\n                TimeUnit.MILLISECONDS.sleep(100);\n            } catch (TimeoutException e) {\n                throw new RuntimeException(\n                        \"Failed to open state table for function \" + tenant + \"/\" + namespace + \"/\" + name\n                                + \" within timeout period\", e);\n            }\n        }\n        throw new IOException(\"Failed to open state table for function \" + tenant + \"/\" + namespace + \"/\" + name);\n    }\n\n    @SuppressWarnings(\"unchecked\")\n    @Override\n    public <T extends StateStore> T getStateStore(String tenant, String namespace, String name) throws Exception {\n        // we defer creation of the state table until a java instance is running here.\n        createStateTable(stateStorageServiceUrl, tenant, namespace, name);\n        Table<ByteBuf, ByteBuf> table = openStateTable(tenant, namespace, name);\n        return (T) new BKStateStoreImpl(tenant, namespace, name, table);\n    }\n\n    @Override\n    public void cleanUp(String tenant, String namespace, String name) throws Exception {","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java#L176-L212","documentation":"BKStateStoreProviderImpl.openStateTable throws RuntimeException on TimeoutException while opening the function's state table via the table service within the retry/timeout budget (retrying every 100 ms on internal server errors). If the table still cannot be opened after the loop, it also throws an IOException with a similar message; this error specifically surfaces the timeout variant. The table service could not serve an open-able view of the table in time.","triggerScenarios":"openStateTable (called via table(), used by getStateStore) repeatedly hits InternalServerError (retried with 100 ms sleeps) until a TimeoutException terminates the loop, or the underlying open times out directly.","commonSituations":"BookKeeper table service (proxy/storage) overloaded or restarting; table recovering/rebalancing during broker failover; state table owned by an unavailable bookie; DNS/network issues from the function instance to the storage proxy.","solutions":["Inspect the TimeoutException cause and preceding internal-server-error logs from the table service.","Check BookKeeper storage cluster health (bookies up, proxy reachable) and wait/retry — table recovery may complete shortly.","Verify network/DNS connectivity from the function instance to the table service proxy.","Restart the function instance after the storage cluster stabilizes; if persistent, investigate the table's bookie ensemble."],"exampleFix":"// before\nTable<ByteBuf, ByteBuf> table = provider.getStateStore(...); // blocks, may time out once\n// after\nTable<ByteBuf, ByteBuf> table = retryWithBackoff(3, () -> provider.getStateStore(...)); // tolerate transient table-service recovery","handlingStrategy":"retry","validationCode":"// check the table service is responsive before opening\n// probe cluster health via broker admin or storage proxy metrics endpoint\nboolean storageHealthy = probeHttp(storageProxyUrl + \"/metrics\");","typeGuard":null,"tryCatchPattern":"try {\n    Table<ByteBuf, ByteBuf> t = openStateTable(tenant, ns, name);\n} catch (RuntimeException e) {\n    if (e.getCause() instanceof TimeoutException || hasInternalServerErrors(e)) {\n        // table service recovering: wait and retry once before failing the instance\n        sleepSeconds(5);\n        return openStateTable(tenant, ns, name);\n    }\n    throw e;\n}","preventionTips":["Keep the BookKeeper table service proxy and bookies healthy and monitored.","Schedule function restarts away from storage maintenance/rebalance windows.","Verify network/DNS from function instances to the storage proxy.","Retry instance startup with backoff — table open often succeeds after recovery."],"tags":["state","bookkeeper","timeout","table-service","runtime-exception"],"backgroundTag":"state-table-open-timeout","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}