apache/pulsar · error · RestException

Broker is forbidden to do read-write operations

Error message

Broker is forbidden to do read-write operations

What it means

HTTP 403 FORBIDDEN thrown by validatePoliciesReadOnlyAccess: the broker's configuration-metadata store reports policies as read-only (PoliciesReadOnly), so this broker refuses all read-write admin operations against policies. Typically enabled cluster-wide during maintenance or when the config store is a read-only replica.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java:1152

    }

    protected LocalPoliciesResources getLocalPolicies() {
        return pulsar().getPulsarResources().getLocalPolicies();
    }

    protected IsolationPolicyResources namespaceIsolationPolicies(){
        return namespaceResources().getIsolationPolicies();
    }

    protected DynamicConfigurationResources dynamicConfigurationResources() {
        return pulsar().getPulsarResources().getDynamicConfigResources();
    }

    public void validatePoliciesReadOnlyAccess() {
        try {
            if (namespaceResources().getPoliciesReadOnly()) {
                log.debug("Policies are read-only. Broker cannot do read-write operations");
                throw new RestException(Status.FORBIDDEN, "Broker is forbidden to do read-write operations");
            }
        } catch (Exception e) {
            log.warn().exception(e).log("Unable to fetch read-only policy config");
            throw new RestException(e);
        }
    }

    public CompletableFuture<Void> validatePoliciesReadOnlyAccessAsync() {
        return namespaceResources().getPoliciesReadOnlyAsync().thenAccept(readOnly -> {
            if (readOnly) {
                    log.debug("Policies are read-only. Broker cannot do read-write operations");
                                throw new RestException(Status.FORBIDDEN,
                                        "Broker is forbidden to do read-write operations");
            }
        });
    }

    protected CompletableFuture<Void> hasActiveNamespace(String tenant) {

View on GitHub (pinned to 820761864e)

Solutions

  1. Restore write connectivity to the configuration store (check ZK ensemble health/quorum)
  2. Wait out maintenance mode and confirm read-only flag is cleared before retrying
  3. Point the broker at the writable config-store ensemble (correct zookeeperServers configuration)
  4. Retry the admin operation once validatePoliciesReadOnlyAccess no longer reports read-only

Example fix

// before: 403 during read-only window
admin.namespaces().setRetention("public/default", new RetentionPolicies(7, 1024));
// after: gate the operation
if (!admin.brokers().getRuntimeConfiguration().containsKey("policiesReadOnly")) {
    admin.namespaces().setRetention("public/default", new RetentionPolicies(7, 1024));
}
Defensive patterns

Strategy: try-catch

Try / catch

try {
    mutatingAdminCall();
} catch (PulsarAdminException e) {
    if (e.getStatusCode() == 403 && e.getMessage().contains("read-write operations")) {
        // defer the change until the config store is writable again
        scheduleRetryAfterMaintenance();
    } else throw e;
}

Prevention

When it happens

Trigger: Any mutating admin API (namespace policies, tenant changes, cluster updates) while zooKeeperSessionExpireTime/policies read-only mode is active — commonly when the broker lost write access to the config store or read-only mode was explicitly enabled.

Common situations: Broker connected to a read-only ZooKeeper observer/replica; planned maintenance where read-only mode was switched on; config-store quorum loss causing the broker to fall back to read-only behavior.

Understand the failure class

Related errors


AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/7faf19d27c5dbaf4. Report an issue: GitHub.