{"record":{"id":"4fd539e1e8bcaaa0","repo":"apache/rocketmq","slug":"failed-to-load-rocksdb-for-auth-acl-please-check","errorCode":null,"errorMessage":"Failed to load rocksdb for auth_acl, please check whether it is occupied.","messagePattern":"Failed to load rocksdb for auth_acl, please check whether it is occupied\\.","errorType":"console","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"auth/src/main/java/org/apache/rocketmq/auth/authorization/provider/LocalAuthorizationMetadataProvider.java","lineNumber":60,"sourceCode":"import org.apache.rocketmq.common.thread.ThreadPoolMonitor;\nimport org.rocksdb.RocksDB;\n\npublic class LocalAuthorizationMetadataProvider implements AuthorizationMetadataProvider {\n\n    private final static String AUTH_METADATA_COLUMN_FAMILY = new String(RocksDB.DEFAULT_COLUMN_FAMILY,\n        StandardCharsets.UTF_8);\n\n    private ConfigRocksDBStorage storage;\n\n    private LoadingCache<String, Acl> aclCache;\n\n    protected ThreadPoolExecutor cacheRefreshExecutor;\n\n    @Override\n    public void initialize(AuthConfig authConfig, Supplier<?> metadataService) {\n        this.storage = ConfigRocksDBStorage.getStore(authConfig.getAuthConfigPath() + File.separator + \"acls\", false);\n        if (!this.storage.start()) {\n            throw new RuntimeException(\"Failed to load rocksdb for auth_acl, please check whether it is occupied.\");\n        }\n        this.cacheRefreshExecutor = ThreadPoolMonitor.createAndMonitor(\n            1,\n            1,\n            1000 * 60,\n            TimeUnit.MILLISECONDS,\n            \"AclCacheRefresh\",\n            100000\n        );\n\n        this.aclCache = Caffeine.newBuilder()\n            .maximumSize(authConfig.getAclCacheMaxNum())\n            .expireAfterAccess(authConfig.getAclCacheExpiredSecond(), TimeUnit.SECONDS)\n            .refreshAfterWrite(authConfig.getAclCacheRefreshSecond(), TimeUnit.SECONDS)\n            .executor(cacheRefreshExecutor)\n            .build(new AclCacheLoader(this.storage));\n    }\n","sourceCodeStart":42,"sourceCodeEnd":78,"githubUrl":"https://github.com/apache/rocketmq/blob/293f5885719fc4aa3619446a1900f58ccfcfdd29/auth/src/main/java/org/apache/rocketmq/auth/authorization/provider/LocalAuthorizationMetadataProvider.java#L42-L78","documentation":"LocalAuthorizationMetadataProvider.initialize() fails when ConfigRocksDBStorage.getStore(.../acls).start() returns false, meaning the RocksDB directory for ACL metadata cannot be opened. RocksDB takes an exclusive file lock (LOCK file) on its directory, so a second opener — another live broker process, a leftover process, or a stale lock with wrong permissions — causes startup to abort with this RuntimeException.","triggerScenarios":"Starting a second broker instance pointing at the same auth config path (same acls RocksDB directory); a previous broker process still holding the LOCK file after an unclean shutdown; insufficient filesystem permissions or read-only disk on the store path; NFS-mounted config dir where POSIX locks misbehave.","commonSituations":"Running two broker processes during a migration or a debug session with a duplicated store path; container restart where the old process lingered; running broker as different users so the LOCK file is not removable; auth config path shared between broker and controller.","solutions":["Check for another running process holding the store: lsof +D <authConfigPath>/acls or fuser on the directory, and stop the duplicate broker","If the previous shutdown was unclean and no process is alive, remove the stale LOCK file inside the acls RocksDB directory","Verify the broker process user has read/write permission on the whole auth config path and that it is not on a read-only or NFS mount","Give each broker instance its own auth config directory instead of sharing one"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// pre-start check: refuse to launch if another process holds the store\nFile lockFile = new File(authConfigPath, \"acls/LOCK\");\ntry (FileChannel ch = FileChannel.open(lockFile.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE);\n     FileLock lock = ch.tryLock()) {\n    if (lock == null) throw new IllegalStateException(\"acls store already locked by another process\");\n} catch (IOException e) { throw new IllegalStateException(\"acls store not writable\", e); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run exactly one broker process per auth config directory","Automate stale-LOCK cleanup in broker start scripts after verifying no live process","Keep the auth config path on a local (non-NFS) filesystem with write permission for the broker user"],"tags":["auth","rocksdb","file-lock","startup","rocketmq"],"backgroundTag":null,"analyzedSha":"293f5885719fc4aa3619446a1900f58ccfcfdd29","analyzedAt":"2026-08-14T11:50:13.822Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}