apache/rocketmq · critical · RuntimeException

Failed to load rocksdb for auth_acl, please check whether it

Error message

Failed to load rocksdb for auth_acl, please check whether it is occupied.

What it means

LocalAuthorizationMetadataProvider.initialize() fails when ConfigRocksDBStorage.getStore(.../acls).start() returns false, meaning the RocksDB directory for ACL metadata cannot be opened. RocksDB takes an exclusive file lock (LOCK file) on its directory, so a second opener — another live broker process, a leftover process, or a stale lock with wrong permissions — causes startup to abort with this RuntimeException.

Source

Thrown at auth/src/main/java/org/apache/rocketmq/auth/authorization/provider/LocalAuthorizationMetadataProvider.java:60

import org.apache.rocketmq.common.thread.ThreadPoolMonitor;
import org.rocksdb.RocksDB;

public class LocalAuthorizationMetadataProvider implements AuthorizationMetadataProvider {

    private final static String AUTH_METADATA_COLUMN_FAMILY = new String(RocksDB.DEFAULT_COLUMN_FAMILY,
        StandardCharsets.UTF_8);

    private ConfigRocksDBStorage storage;

    private LoadingCache<String, Acl> aclCache;

    protected ThreadPoolExecutor cacheRefreshExecutor;

    @Override
    public void initialize(AuthConfig authConfig, Supplier<?> metadataService) {
        this.storage = ConfigRocksDBStorage.getStore(authConfig.getAuthConfigPath() + File.separator + "acls", false);
        if (!this.storage.start()) {
            throw new RuntimeException("Failed to load rocksdb for auth_acl, please check whether it is occupied.");
        }
        this.cacheRefreshExecutor = ThreadPoolMonitor.createAndMonitor(
            1,
            1,
            1000 * 60,
            TimeUnit.MILLISECONDS,
            "AclCacheRefresh",
            100000
        );

        this.aclCache = Caffeine.newBuilder()
            .maximumSize(authConfig.getAclCacheMaxNum())
            .expireAfterAccess(authConfig.getAclCacheExpiredSecond(), TimeUnit.SECONDS)
            .refreshAfterWrite(authConfig.getAclCacheRefreshSecond(), TimeUnit.SECONDS)
            .executor(cacheRefreshExecutor)
            .build(new AclCacheLoader(this.storage));
    }

View on GitHub (pinned to 293f588571)

Solutions

Check for another running process holding the store: lsof +D <authConfigPath>/acls or fuser on the directory, and stop the duplicate broker
If the previous shutdown was unclean and no process is alive, remove the stale LOCK file inside the acls RocksDB directory
Verify the broker process user has read/write permission on the whole auth config path and that it is not on a read-only or NFS mount
Give each broker instance its own auth config directory instead of sharing one

Defensive patterns

Strategy: validation

Validate before calling

// pre-start check: refuse to launch if another process holds the store
File lockFile = new File(authConfigPath, "acls/LOCK");
try (FileChannel ch = FileChannel.open(lockFile.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
     FileLock lock = ch.tryLock()) {
    if (lock == null) throw new IllegalStateException("acls store already locked by another process");
} catch (IOException e) { throw new IllegalStateException("acls store not writable", e); }

Prevention

Run exactly one broker process per auth config directory
Automate stale-LOCK cleanup in broker start scripts after verifying no live process
Keep the auth config path on a local (non-NFS) filesystem with write permission for the broker user

When it happens

Trigger: Starting a second broker instance pointing at the same auth config path (same acls RocksDB directory); a previous broker process still holding the LOCK file after an unclean shutdown; insufficient filesystem permissions or read-only disk on the store path; NFS-mounted config dir where POSIX locks misbehave.

Common situations: Running two broker processes during a migration or a debug session with a duplicated store path; container restart where the old process lingered; running broker as different users so the LOCK file is not removable; auth config path shared between broker and controller.

Related errors

AI-assisted analysis of apache/rocketmq@293f588571 (2026-08-14). Data as JSON: /api/errors/4fd539e1e8bcaaa0. Report an issue: GitHub.