tikv/tikv · error

failed to get either modify_lock or client

Error message

failed to get either modify_lock or client

What it means

In get_client, the code attempts to acquire the modify_lock to build/refresh a token-authenticated BlobServiceClient; if the lock is poisoned (a previous holder panicked while holding the lock) or the lock request fails, neither a client nor the lock can be obtained and this InvalidInput error is returned.

Source

Thrown at components/cloud/azure/src/azblob.rs:541

            let scopes = vec![&self.token_resource as &str];
            let token =
                self.token_cred.get_token(&scopes).await.map_err(|e| {
                    io::Error::new(io::ErrorKind::InvalidInput, format!("{:?}", &e))
                })?;
            let blob_service = BlobServiceClient::new(
                self.account_name.clone(),
                StorageCredentials::bearer_token(token.token.secret().to_string()),
            );
            let storage_client =
                Arc::new(blob_service.container_client(self.container_name.clone()));

            {
                let mut token_response = self.token_cache.write().unwrap();
                *token_response = Some((token, storage_client.clone()));
            }
            Ok(storage_client)
        } else {
            Err(io::Error::new(
                io::ErrorKind::InvalidInput,
                "failed to get either modify_lock or client",
            ))
        }
    }
}

const STORAGE_NAME: &str = "azure";

#[derive(Clone)]
pub struct AzureStorage {
    config: Config,
    client_builder: Arc<dyn ContainerBuilder>,
}

impl AzureStorage {
    pub fn from_input(input: InputConfig) -> io::Result<Self> {
        Self::new(Config::from_input(input)?)

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Fix the original panic that poisoned modify_lock — check logs for the earlier panic in the token refresh path.
2. Restart the affected TiKV/BR process to recreate the lock state.
3. Harden the critical section to avoid panics (no unwrap on token_cache entries).

Example fix

// before: panics propagate and poison the lock
let mut token_response = self.token_cache.write().unwrap();
// after: recover or fail cleanly without poisoning on every call
match self.modify_lock.write() {
    Ok(mut guard) => { /* refresh */ },
    Err(poisoned) => return Err(io::Error::new(io::ErrorKind::Other, "token cache poisoned, restart required")),
}

// recover from poisoned lock instead of failing forever
let client = match self.modify_lock.write() {
    Ok(g) => build_client(g),
    Err(poisoned) => {
        log::warn!("modify_lock poisoned; recovering");
        build_client(poisoned.into_inner())
    }
};

When it happens

Trigger: Calling get_client with Azure AD token credentials when self.modify_lock.write() fails — i.e. the RwLock is poisoned because another thread panicked while updating token_cache.

Common situations: A prior panic inside the token refresh critical section (e.g. unwrap on poisoned cache) leaving the RwLock poisoned; subsequent all calls to get_client then fail with this message on every retry.

Related errors

• unexpected state {}
• unexpcted state {}
• txn record found but not expected: {:?} {} {:?} {:?} [region
• account name cannot be empty to access azure blob storage
• {:?}

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/28748964e8076bab. Report an issue: GitHub.