{"record":{"id":"f60a7a2d61bcf0e6","repo":"databendlabs/databend","slug":"verifier-client-error","errorCode":null,"errorMessage":"verifier client error: {}","messagePattern":"verifier client error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/binaries/metaverifier/main.rs","lineNumber":212,"sourceCode":"                }\n            }\n            tx.send(()).unwrap();\n\n            Ok(())\n        },\n        None,\n    );\n\n    let _ret = wait_verifier_handle.await.unwrap();\n    if let Err(e) = waiter_handle.await {\n        let end = Instant::now();\n        println!(\n            \"verifier client return error: {:?} in {} milliseconds\",\n            e,\n            end.duration_since(start).as_millis()\n        );\n        fs::write(VERIFIER_RESULT_FILE, \"ERROR\")?;\n        bail!(\"verifier client error: {}\", e);\n    }\n\n    let end = Instant::now();\n    println!(\n        \"verifier client({}) * number({}) in {} milliseconds\",\n        config.client,\n        config.number,\n        end.duration_since(start).as_millis()\n    );\n\n    // write a file as end\n    fs::write(VERIFIER_RESULT_FILE, \"END\")?;\n\n    Ok(())\n}\n\nasync fn verifier(\n    client: &Arc<ClientHandle<DatabendRuntime>>,","sourceCodeStart":194,"sourceCodeEnd":230,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/binaries/metaverifier/main.rs#L194-L230","documentation":"The verifier's client operations (put/get/remove of meta keys) returned an error from the meta gRPC API. metaverifier logs the elapsed time, writes \"ERROR\" to the result file, and bails with `verifier client error: {e}`. It indicates a failure during the actual verification RPCs, not during client setup.","triggerScenarios":"Running metaverifier against a live meta cluster where an RPC fails: timeouts, meta service restarting mid-run, permission/quota errors, serialization mismatch between verifier and server API versions.","commonSituations":"Meta service under load or being redeployed while verification runs; network partition during a long verification; metaverifier built from a version incompatible with the meta cluster's API.","solutions":["Inspect the printed inner error (`verifier client return error: ...`) for the root cause (timeout vs refused vs API error).","Ensure the meta cluster is stable and reachable for the full verification duration; rerun.","Match metaverifier and databend-meta versions/builds to avoid API incompatibility.","Increase timeouts / reduce client count in metaverifier config if the cluster is overloaded."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// verify cluster health before a long verification run\nlet healthy = client.health().await.is_ok();\nif !healthy {\n    return Err(anyhow::anyhow!(\"meta cluster unhealthy; aborting verification before start\"));\n}","typeGuard":null,"tryCatchPattern":"match verifier(&client, prefix, ...).await {\n    Err(e) if is_retryable(&e) => {\n        tokio::time::sleep(Duration::from_secs(2)).await;\n        verifier(&client, prefix, ...).await\n    }\n    Err(e) => {\n        fs::write(VERIFIER_RESULT_FILE, \"ERROR\")?;\n        bail!(\"verifier client error: {}\", e)\n    }\n    ok => ok,\n}","preventionTips":["Run verification during a quiet window; avoid meta restarts mid-run.","Add per-RPC timeouts and retries on transient errors.","Pin matching versions of metaverifier and databend-meta.","Check the printed inner error (`verifier client return error`) to classify timeout vs API failure before retrying."],"tags":["metaverifier","grpc","rpc","verification"],"backgroundTag":"api-error-response","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}