{"record":{"id":"286029609884b796","repo":"apache/pulsar","slug":"policies-are-in-readonly-mode","errorCode":null,"errorMessage":"policies are in readonly mode","messagePattern":"policies are in readonly mode","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java","lineNumber":239,"sourceCode":"                if (conf.isAuthorizationAllowWildcardsMatching()) {\n                    if (checkWildcardPermission(role, authAction, namespaceRoles)) {\n                        // The role has namespace level permission by wildcard match\n                        return true;\n                    }\n                }\n            }\n            return false;\n        });\n    }\n\n    @Override\n    public CompletableFuture<Void> grantPermissionAsync(TopicName topicName, Set<AuthAction> actions,\n                                                        String role, String authDataJson) {\n\n        return getPoliciesReadOnlyAsync().thenCompose(readonly -> {\n            if (readonly) {\n                log.debug(\"Policies are read-only. Broker cannot do read-write operations\");\n                throw new IllegalStateException(\"policies are in readonly mode\");\n            }\n            String topicUri = topicName.toString();\n            return pulsarResources.getNamespaceResources()\n                    .setPoliciesAsync(topicName.getNamespaceObject(), policies -> {\n                        policies.auth_policies.getTopicAuthentication()\n                                .computeIfAbsent(topicUri, __ -> new HashMap<>())\n                                .put(role, actions);\n                        return policies;\n                    }).whenComplete((__, ex) -> {\n                        if (ex != null) {\n                            log.error()\n                                    .attr(\"role\", role)\n                                    .attr(\"topic\", topicName)\n                                    .exception(ex)\n                                    .log(\"Failed to set permissions for role on topic\");\n                        } else {\n                            log.info()\n                                    .attr(\"role\", role)","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java#L221-L257","documentation":"grantPermissionAsync(TopicName, Set<AuthAction>, String, String) first checks getPoliciesReadOnlyAsync; when the broker is in policies read-only mode it throws IllegalStateException because policy writes are disallowed (e.g. the broker lost metadata-store write access or is a read-only replica).","triggerScenarios":"Invoking grant-permission admin APIs while the broker reports policies read-only — typically when the local ZooKeeper/metadata store is unreachable for writes, or the broker is configured as read-only for policy updates.","commonSituations":"ZooKeeper quorum degraded or in read-only state during maintenance; network partition between broker and metadata store; running the admin command against a broker that cannot write policies instead of the active one.","solutions":["Check broker logs for the paired debug/log 'Policies are read-only' and the underlying metadata-store connectivity errors.","Restore ZooKeeper/metadata-store health and connectivity, then retry the grant.","Verify you're issuing the write against a broker with write access to the configuration store.","After connectivity is restored, confirm read-only mode cleared (broker reloads policies) before retrying."],"exampleFix":"// before\nadmin.topicPolicies(\"persistent://tenant/ns/topic\").grantPermission(role, EnumSet.of(AuthAction.produce)); // broker read-only\n// after\n// ensure zk / metadata store healthy first, then retry\nawaitBrokerPoliciesWritable();\nadmin.topicPolicies(\"persistent://tenant/ns/topic\").grantPermission(role, EnumSet.of(AuthAction.produce));","handlingStrategy":"retry","validationCode":"// check policies writable before calling\nboolean writable = !brokerStatus.isPoliciesReadOnly(); // or probe with a harmless metadata read/write","typeGuard":"boolean canWritePolicies(AuthorizationProvider p) { return !p.getPoliciesReadOnlyAsync().join(); }","tryCatchPattern":"try { provider.grantPermissionAsync(topic, actions, role, authJson).join(); } catch (CompletionException e) { if (e.getCause() instanceof IllegalStateException && e.getCause().getMessage().contains(\"readonly\")) { backoffAndRetry(); } else throw e; }","preventionTips":["Schedule policy writes outside metadata-store maintenance windows","Monitor ZooKeeper quorum health and alert on read-only state","Route admin writes to brokers with confirmed write access","Implement bounded backoff-retry for policy mutations"],"tags":["authorization","policies","readonly","metadata-store"],"backgroundTag":"policies-readonly-mode","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}