{"record":{"id":"7faf19d27c5dbaf4","repo":"apache/pulsar","slug":"broker-is-forbidden-to-do-read-write-operations-7faf19","errorCode":null,"errorMessage":"Broker is forbidden to do read-write operations","messagePattern":"Broker is forbidden to do read-write operations","errorType":"http","errorClass":"RestException","httpStatus":403,"severity":"error","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java","lineNumber":1152,"sourceCode":"    }\n\n    protected LocalPoliciesResources getLocalPolicies() {\n        return pulsar().getPulsarResources().getLocalPolicies();\n    }\n\n    protected IsolationPolicyResources namespaceIsolationPolicies(){\n        return namespaceResources().getIsolationPolicies();\n    }\n\n    protected DynamicConfigurationResources dynamicConfigurationResources() {\n        return pulsar().getPulsarResources().getDynamicConfigResources();\n    }\n\n    public void validatePoliciesReadOnlyAccess() {\n        try {\n            if (namespaceResources().getPoliciesReadOnly()) {\n                log.debug(\"Policies are read-only. Broker cannot do read-write operations\");\n                throw new RestException(Status.FORBIDDEN, \"Broker is forbidden to do read-write operations\");\n            }\n        } catch (Exception e) {\n            log.warn().exception(e).log(\"Unable to fetch read-only policy config\");\n            throw new RestException(e);\n        }\n    }\n\n    public CompletableFuture<Void> validatePoliciesReadOnlyAccessAsync() {\n        return namespaceResources().getPoliciesReadOnlyAsync().thenAccept(readOnly -> {\n            if (readOnly) {\n                    log.debug(\"Policies are read-only. Broker cannot do read-write operations\");\n                                throw new RestException(Status.FORBIDDEN,\n                                        \"Broker is forbidden to do read-write operations\");\n            }\n        });\n    }\n\n    protected CompletableFuture<Void> hasActiveNamespace(String tenant) {","sourceCodeStart":1134,"sourceCodeEnd":1170,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java#L1134-L1170","documentation":"HTTP 403 FORBIDDEN thrown by validatePoliciesReadOnlyAccess: the broker's configuration-metadata store reports policies as read-only (PoliciesReadOnly), so this broker refuses all read-write admin operations against policies. Typically enabled cluster-wide during maintenance or when the config store is a read-only replica.","triggerScenarios":"Any mutating admin API (namespace policies, tenant changes, cluster updates) while zooKeeperSessionExpireTime/policies read-only mode is active — commonly when the broker lost write access to the config store or read-only mode was explicitly enabled.","commonSituations":"Broker connected to a read-only ZooKeeper observer/replica; planned maintenance where read-only mode was switched on; config-store quorum loss causing the broker to fall back to read-only behavior.","solutions":["Restore write connectivity to the configuration store (check ZK ensemble health/quorum)","Wait out maintenance mode and confirm read-only flag is cleared before retrying","Point the broker at the writable config-store ensemble (correct zookeeperServers configuration)","Retry the admin operation once validatePoliciesReadOnlyAccess no longer reports read-only"],"exampleFix":"// before: 403 during read-only window\nadmin.namespaces().setRetention(\"public/default\", new RetentionPolicies(7, 1024));\n// after: gate the operation\nif (!admin.brokers().getRuntimeConfiguration().containsKey(\"policiesReadOnly\")) {\n    admin.namespaces().setRetention(\"public/default\", new RetentionPolicies(7, 1024));\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    mutatingAdminCall();\n} catch (PulsarAdminException e) {\n    if (e.getStatusCode() == 403 && e.getMessage().contains(\"read-write operations\")) {\n        // defer the change until the config store is writable again\n        scheduleRetryAfterMaintenance();\n    } else throw e;\n}","preventionTips":["Check broker read-only status before running policy automation during maintenance","Monitor config-store quorum health to anticipate read-only fallback","Sequence admin changes to avoid maintenance windows"],"tags":["http-403","read-only","configuration-store","maintenance"],"backgroundTag":"config-store-read-only","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}