apache/pulsar · error · IOException

Failed to setup / verify state table for function %s/%s/%s w

Error message

Failed to setup / verify state table for function %s/%s/%s within timeout

What it means

BKStateStoreProviderImpl.createStateTable throws IOException after exhausting its retry loop while trying to set up or verify the function's BookKeeper-backed state table. During the loop it re-attempts every 100 ms on metadata-fetch issues, recording lastException; when the timeout budget is spent it fails with this message. The state table could not be created/verified before the instance could start using state.

Source

Thrown at pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/BKStateStoreProviderImpl.java:162

                    } catch (Exception e) {
                        // there might be two client conflicting at creating table, so let's retrieve it to make
                        // sure the table is created.
                        lastException = e;
                        log.warn()
                                .attr("tableNs", tableNs)
                                .attr("tableName", tableName)
                                .exception(e)
                                .log("Encountered exception when creating table");
                    }
                } catch (ClientException ce) {
                    log.warn()
                            .attr("message", ce.getMessage())
                            .log("Encountered issue on fetching state stable metadata,"
                                    + " re-attempting in 100 milliseconds");
                    TimeUnit.MILLISECONDS.sleep(100);
                }
            }
            throw new IOException(
                    String.format("Failed to setup / verify state table for function %s/%s/%s within timeout", tenant,
                            name, name), lastException);
        }
    }

    private Table<ByteBuf, ByteBuf> openStateTable(String tenant,
                                                   String namespace,
                                                   String name) throws Exception {
        StorageClient client = getStorageClient(tenant, namespace);

        log.info()
                .attr("tenant", tenant)
                .attr("namespace", namespace)
                .attr("name", name)
                .log("Opening state table for function");
        // NOTE: this is a workaround until we bump bk version to 4.9.0
        // table might just be created above, so it might not be ready for serving traffic
        Stopwatch openSw = Stopwatch.createStarted();

View on GitHub (pinned to 820761864e)

Solutions

1. Read lastException (cause) to see the repeated underlying metadata error.
2. Verify the function's namespace exists and state is enabled with correct bookkeeper_auth/bookkeeper_ack_quorum settings in namespace policies.
3. Check admin client credentials/role have namespace-admin permissions.
4. Verify broker/service URL connectivity; increase the state setup timeout or retry after the cluster recovers.

Example fix

// before
// state not configured on namespace -> repeated fetch failures
// after
bin/pulsar-admin namespaces set-persistence tenant/ns \
  --bookkeeper-ack-quorum 2 --bookkeeper-write-quorum 2 --bookkeeper-envelope-key false
# ensure admin role can manage the namespace, then restart the function instance

// before deploying the function with state enabled
NamespacePolicies p = admin.namespaces().getPolicies(tenant + "/" + namespace);
// state persistence must be configured
if (p.getPersistence() == null || p.getPersistence().getBookkeeperAckQuorum() < 1) {
    throw new IllegalStateException("state persistence not configured on namespace");
}
admin.namespaces().getPermissions(tenant + "/" + namespace) // confirm admin role access

try {
    StateStore store = provider.getStateStore(tenant, ns, name, /*...*/);
} catch (IOException e) {
    // message contains tenant/name; cause (lastException) shows the repeated failure
    log.error("state table setup failed for {}/{}: {}", tenant, name, e.getCause(), e);
    throw e; // fail fast; instance restart will re-attempt setup
}

When it happens

Trigger: createStateTable (invoked from getStateStore) cannot fetch or create the table metadata within the timeout window: repeated PulsarAdmin/namespace errors, permission denials on state namespace policies, or persistent metadata fetch failures.

Common situations: Missing namespace-level state configuration (bookkeeper authentication/idempotency settings not applied); admin client misconfigured (wrong service URL/credentials); namespace policies API failing; long broker unavailability exceeding the retry window.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to open state table for function ${tenant}/${namespac
• Timeout during managed ledger close
• Timeout during managed ledger delete operation
• Failed to increment key '%s' by amount '%s'
• Failed to retrieve counter from key '%s'

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/b2ab540a37d15e1c. Report an issue: GitHub.