What causes "Concurrent modification" in apache/pulsar?

Retry the grant: re-read current permissions with GET /namespaces/{ns}/permissions, re-apply the desired diff, and resubmit.

apache/pulsar · warning · RestException

Concurrent modification

Error message

Concurrent modification

What it means

HTTP 409 returned by the namespace permission-grant admin API when the underlying metadata store reports an optimistic-concurrency failure (MetadataStoreException.BadVersionException) or an IllegalStateException propagated by legacy code. The permissions policy was modified concurrently by another broker/admin operation, so the write was rejected instead of silently overwriting.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java:681

                .exceptionally(ex -> {
                    Throwable realCause = FutureUtil.unwrapCompletionException(ex);
                    //The IllegalArgumentException and the IllegalStateException were historically thrown by the
                    // grantPermissionAsync method, so we catch them here to ensure backwards compatibility.
                    if (realCause instanceof MetadataStoreException.NotFoundException
                            || realCause instanceof IllegalArgumentException) {
                        log.warn()
                                .attr("namespace", namespaceName)
                                .exception(ex)
                                .log("Failed to set permissions for namespace : does not exist");
                        throw new RestException(Status.NOT_FOUND, "Topic's namespace does not exist");
                    } else if (realCause instanceof MetadataStoreException.BadVersionException
                            || realCause instanceof IllegalStateException) {
                        log.warn()
                                .attr("namespace", namespaceName)
                                .exceptionMessage(ex.getCause())
                                .exception(ex)
                                .log("Failed to set permissions for namespace");
                        throw new RestException(Status.CONFLICT, "Concurrent modification");
                    } else {
                        log.error()
                                .attr("namespace", namespaceName)
                                .exception(ex)
                                .log("Failed to get permissions for namespace");
                        throw new RestException(realCause);
                    }
                });
    }

    protected CompletableFuture<Void> internalGrantPermissionOnTopicsAsync(List<GrantTopicPermissionOptions> options) {
        return checkNamespace(options.stream().map(o -> TopicName.get(o.getTopic()).getNamespace()))
                .thenCompose(__ -> validateAdminAccessForTenantAsync(
                        TopicName.get(options.get(0).getTopic()).getTenant())
                ).thenCompose(__ -> internalCheckTopicExists(options.stream().map(o -> TopicName.get(o.getTopic()))))
                .thenCompose(__ -> getAuthorizationService().grantPermissionAsync(options))
                .thenAccept(unused -> log.info()
                        .attr("options", options)

View on GitHub (pinned to 820761864e)

Solutions

Retry the grant: re-read current permissions with GET /namespaces/{ns}/permissions, re-apply the desired diff, and resubmit.
Serialize permission changes for a namespace (single admin process, lock, or queue) instead of concurrent updates.
Inspect broker logs (WARN 'Failed to set permissions for namespace') to identify the competing operation.
Upgrade Pulsar if you see IllegalStateException without a real concurrent writer — legacy error propagation path.

Example fix

// before: fire-and-forget concurrent grant
twoAdmins.forEach(a -> a.grantPermission(ns, role, actions));
// after: retry on 409
retryOnConflict(3, () -> grantPermission(ns, role, actions));

Defensive patterns

Strategy: retry

Validate before calling

// no reliable pre-check; minimize the window instead
List<GrantedPermission> current = admin.namespaces().getPermissions(ns);

Try / catch

try {
    admin.namespaces().grantPermissionOnNamespace(ns, role, actions);
} catch (PulsarAdminException.ConflictException e) {
    // re-read and retry with backoff
    retry(3, () -> admin.namespaces().grantPermissionOnNamespace(ns, role, actions));
}

Prevention

Use a single writer (or distributed lock) per namespace for policy changes
Re-read current permissions immediately before each grant
Retry 409s with exponential backoff instead of failing fast
Avoid parallel CI jobs mutating the same namespace's ACLs

When it happens

Trigger: Calling POST/PUT on /namespaces/{tenant}/{namespace}/permissions/{role} (internalGrantPermissionOnNamespaceAsync) while another admin client or broker updates the same namespace's permission policies; the metadata store version check fails, or a wrapped IllegalStateException surfaces from grantPermissionAsync.

Common situations: Two admins or automation scripts granting/revoking permissions on the same namespace at once; configuration-management tools (Terraform/Ansible) racing; repeated rapid permission updates through the REST API hitting the same z-node/etcd key version.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/f33a54ef85d24bfc. Report an issue: GitHub.