{"record":{"id":"b2ab540a37d15e1c","repo":"apache/pulsar","slug":"failed-to-setup-verify-state-table-for-function","errorCode":null,"errorMessage":"Failed to setup / verify state table for function %s/%s/%s within timeout","messagePattern":"Failed to setup / verify state table for function (.+?)/(.+?)/(.+?) within timeout","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java","lineNumber":162,"sourceCode":"                    } catch (Exception e) {\n                        // there might be two client conflicting at creating table, so let's retrieve it to make\n                        // sure the table is created.\n                        lastException = e;\n                        log.warn()\n                                .attr(\"tableNs\", tableNs)\n                                .attr(\"tableName\", tableName)\n                                .exception(e)\n                                .log(\"Encountered exception when creating table\");\n                    }\n                } catch (ClientException ce) {\n                    log.warn()\n                            .attr(\"message\", ce.getMessage())\n                            .log(\"Encountered issue on fetching state stable metadata,\"\n                                    + \" re-attempting in 100 milliseconds\");\n                    TimeUnit.MILLISECONDS.sleep(100);\n                }\n            }\n            throw new IOException(\n                    String.format(\"Failed to setup / verify state table for function %s/%s/%s within timeout\", tenant,\n                            name, name), lastException);\n        }\n    }\n\n    private Table<ByteBuf, ByteBuf> openStateTable(String tenant,\n                                                   String namespace,\n                                                   String name) throws Exception {\n        StorageClient client = getStorageClient(tenant, namespace);\n\n        log.info()\n                .attr(\"tenant\", tenant)\n                .attr(\"namespace\", namespace)\n                .attr(\"name\", name)\n                .log(\"Opening state table for function\");\n        // NOTE: this is a workaround until we bump bk version to 4.9.0\n        // table might just be created above, so it might not be ready for serving traffic\n        Stopwatch openSw = Stopwatch.createStarted();","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java#L144-L180","documentation":"BKStateStoreProviderImpl.createStateTable throws IOException after exhausting its retry loop while trying to set up or verify the function's BookKeeper-backed state table. During the loop it re-attempts every 100 ms on metadata-fetch issues, recording lastException; when the timeout budget is spent it fails with this message. The state table could not be created/verified before the instance could start using state.","triggerScenarios":"createStateTable (invoked from getStateStore) cannot fetch or create the table metadata within the timeout window: repeated PulsarAdmin/namespace errors, permission denials on state namespace policies, or persistent metadata fetch failures.","commonSituations":"Missing namespace-level state configuration (bookkeeper authentication/idempotency settings not applied); admin client misconfigured (wrong service URL/credentials); namespace policies API failing; long broker unavailability exceeding the retry window.","solutions":["Read lastException (cause) to see the repeated underlying metadata error.","Verify the function's namespace exists and state is enabled with correct bookkeeper_auth/bookkeeper_ack_quorum settings in namespace policies.","Check admin client credentials/role have namespace-admin permissions.","Verify broker/service URL connectivity; increase the state setup timeout or retry after the cluster recovers."],"exampleFix":"// before\n// state not configured on namespace -> repeated fetch failures\n// after\nbin/pulsar-admin namespaces set-persistence tenant/ns \\\n  --bookkeeper-ack-quorum 2 --bookkeeper-write-quorum 2 --bookkeeper-envelope-key false\n# ensure admin role can manage the namespace, then restart the function instance","handlingStrategy":"validation","validationCode":"// before deploying the function with state enabled\nNamespacePolicies p = admin.namespaces().getPolicies(tenant + \"/\" + namespace);\n// state persistence must be configured\nif (p.getPersistence() == null || p.getPersistence().getBookkeeperAckQuorum() < 1) {\n    throw new IllegalStateException(\"state persistence not configured on namespace\");\n}\nadmin.namespaces().getPermissions(tenant + \"/\" + namespace) // confirm admin role access","typeGuard":null,"tryCatchPattern":"try {\n    StateStore store = provider.getStateStore(tenant, ns, name, /*...*/);\n} catch (IOException e) {\n    // message contains tenant/name; cause (lastException) shows the repeated failure\n    log.error(\"state table setup failed for {}/{}: {}\", tenant, name, e.getCause(), e);\n    throw e; // fail fast; instance restart will re-attempt setup\n}","preventionTips":["Set namespace persistence/bookkeeper auth policies before enabling function state.","Ensure the function's service account has namespace admin/produce rights.","Verify broker connectivity with pulsar-admin before deploying stateful functions."],"tags":["state","bookkeeper","timeout","pulsar-admin","io-exception"],"backgroundTag":"state-table-setup-timeout","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}