{"record":{"id":"f33a54ef85d24bfc","repo":"apache/pulsar","slug":"concurrent-modification","errorCode":null,"errorMessage":"Concurrent modification","messagePattern":"Concurrent modification","errorType":"http","errorClass":"RestException","httpStatus":409,"severity":"warning","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java","lineNumber":681,"sourceCode":"                .exceptionally(ex -> {\n                    Throwable realCause = FutureUtil.unwrapCompletionException(ex);\n                    //The IllegalArgumentException and the IllegalStateException were historically thrown by the\n                    // grantPermissionAsync method, so we catch them here to ensure backwards compatibility.\n                    if (realCause instanceof MetadataStoreException.NotFoundException\n                            || realCause instanceof IllegalArgumentException) {\n                        log.warn()\n                                .attr(\"namespace\", namespaceName)\n                                .exception(ex)\n                                .log(\"Failed to set permissions for namespace : does not exist\");\n                        throw new RestException(Status.NOT_FOUND, \"Topic's namespace does not exist\");\n                    } else if (realCause instanceof MetadataStoreException.BadVersionException\n                            || realCause instanceof IllegalStateException) {\n                        log.warn()\n                                .attr(\"namespace\", namespaceName)\n                                .exceptionMessage(ex.getCause())\n                                .exception(ex)\n                                .log(\"Failed to set permissions for namespace\");\n                        throw new RestException(Status.CONFLICT, \"Concurrent modification\");\n                    } else {\n                        log.error()\n                                .attr(\"namespace\", namespaceName)\n                                .exception(ex)\n                                .log(\"Failed to get permissions for namespace\");\n                        throw new RestException(realCause);\n                    }\n                });\n    }\n\n    protected CompletableFuture<Void> internalGrantPermissionOnTopicsAsync(List<GrantTopicPermissionOptions> options) {\n        return checkNamespace(options.stream().map(o -> TopicName.get(o.getTopic()).getNamespace()))\n                .thenCompose(__ -> validateAdminAccessForTenantAsync(\n                        TopicName.get(options.get(0).getTopic()).getTenant())\n                ).thenCompose(__ -> internalCheckTopicExists(options.stream().map(o -> TopicName.get(o.getTopic()))))\n                .thenCompose(__ -> getAuthorizationService().grantPermissionAsync(options))\n                .thenAccept(unused -> log.info()\n                        .attr(\"options\", options)","sourceCodeStart":663,"sourceCodeEnd":699,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java#L663-L699","documentation":"HTTP 409 returned by the namespace permission-grant admin API when the underlying metadata store reports an optimistic-concurrency failure (MetadataStoreException.BadVersionException) or an IllegalStateException propagated by legacy code. The permissions policy was modified concurrently by another broker/admin operation, so the write was rejected instead of silently overwriting.","triggerScenarios":"Calling POST/PUT on /namespaces/{tenant}/{namespace}/permissions/{role} (internalGrantPermissionOnNamespaceAsync) while another admin client or broker updates the same namespace's permission policies; the metadata store version check fails, or a wrapped IllegalStateException surfaces from grantPermissionAsync.","commonSituations":"Two admins or automation scripts granting/revoking permissions on the same namespace at once; configuration-management tools (Terraform/Ansible) racing; repeated rapid permission updates through the REST API hitting the same z-node/etcd key version.","solutions":["Retry the grant: re-read current permissions with GET /namespaces/{ns}/permissions, re-apply the desired diff, and resubmit.","Serialize permission changes for a namespace (single admin process, lock, or queue) instead of concurrent updates.","Inspect broker logs (WARN 'Failed to set permissions for namespace') to identify the competing operation.","Upgrade Pulsar if you see IllegalStateException without a real concurrent writer — legacy error propagation path."],"exampleFix":"// before: fire-and-forget concurrent grant\ntwoAdmins.forEach(a -> a.grantPermission(ns, role, actions));\n// after: retry on 409\nretryOnConflict(3, () -> grantPermission(ns, role, actions));","handlingStrategy":"retry","validationCode":"// no reliable pre-check; minimize the window instead\nList<GrantedPermission> current = admin.namespaces().getPermissions(ns);","typeGuard":null,"tryCatchPattern":"try {\n    admin.namespaces().grantPermissionOnNamespace(ns, role, actions);\n} catch (PulsarAdminException.ConflictException e) {\n    // re-read and retry with backoff\n    retry(3, () -> admin.namespaces().grantPermissionOnNamespace(ns, role, actions));\n}","preventionTips":["Use a single writer (or distributed lock) per namespace for policy changes","Re-read current permissions immediately before each grant","Retry 409s with exponential backoff instead of failing fast","Avoid parallel CI jobs mutating the same namespace's ACLs"],"tags":["rest-api","admin","concurrency","metadata-store"],"backgroundTag":"concurrent-modification-conflict","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}