databendlabs/databend · error

verifier client error

Error message

verifier client error: {}

What it means

The verifier's client operations (put/get/remove of meta keys) returned an error from the meta gRPC API. metaverifier logs the elapsed time, writes "ERROR" to the result file, and bails with `verifier client error: {e}`. It indicates a failure during the actual verification RPCs, not during client setup.

Solutions

1. Inspect the printed inner error (`verifier client return error: ...`) for the root cause (timeout vs refused vs API error).
2. Ensure the meta cluster is stable and reachable for the full verification duration; rerun.
3. Match metaverifier and databend-meta versions/builds to avoid API incompatibility.
4. Increase timeouts / reduce client count in metaverifier config if the cluster is overloaded.

// verify cluster health before a long verification run
let healthy = client.health().await.is_ok();
if !healthy {
    return Err(anyhow::anyhow!("meta cluster unhealthy; aborting verification before start"));
}

match verifier(&client, prefix, ...).await {
    Err(e) if is_retryable(&e) => {
        tokio::time::sleep(Duration::from_secs(2)).await;
        verifier(&client, prefix, ...).await
    }
    Err(e) => {
        fs::write(VERIFIER_RESULT_FILE, "ERROR")?;
        bail!("verifier client error: {}", e)
    }
    ok => ok,
}

When it happens

Trigger: Running metaverifier against a live meta cluster where an RPC fails: timeouts, meta service restarting mid-run, permission/quota errors, serialization mismatch between verifier and server API versions.

Common situations: Meta service under load or being redeployed while verification runs; network partition during a long verification; metaverifier built from a version incompatible with the meta cluster's API.

Related errors

• not implemented
• grpc_api_address MUST not be empty!
• Failed to create client
• ContextVar has no value
• remove_percent MUST in [0, 100)!

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/f60a7a2d61bcf0e6. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/binaries/metaverifier/main.rs:212

                }
            }
            tx.send(()).unwrap();

            Ok(())
        },
        None,
    );

    let _ret = wait_verifier_handle.await.unwrap();
    if let Err(e) = waiter_handle.await {
        let end = Instant::now();
        println!(
            "verifier client return error: {:?} in {} milliseconds",
            e,
            end.duration_since(start).as_millis()
        );
        fs::write(VERIFIER_RESULT_FILE, "ERROR")?;
        bail!("verifier client error: {}", e);
    }

    let end = Instant::now();
    println!(
        "verifier client({}) * number({}) in {} milliseconds",
        config.client,
        config.number,
        end.duration_since(start).as_millis()
    );

    // write a file as end
    fs::write(VERIFIER_RESULT_FILE, "END")?;

    Ok(())
}

async fn verifier(
    client: &Arc<ClientHandle<DatabendRuntime>>,

View on GitHub (pinned to 288d84d76e)