{"record":{"id":"32f4c7cbfdd58221","repo":"databendlabs/databend","slug":"no-metasrv-running-on-32f4c7","errorCode":null,"errorMessage":"no metasrv running on: {}","messagePattern":"no metasrv running on: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/control/src/keys_layout_from_grpc.rs","lineNumber":59,"sourceCode":"\n/// if port is open, service is running\nasync fn is_service_running(addr: SocketAddr) -> Result<bool, io::Error> {\n    let socket = TcpSocket::new_v4()?;\n    let stream = socket.connect(addr).await;\n\n    Ok(stream.is_ok())\n}\n\n/// try to get available grpc api socket address\nasync fn get_available_socket_addr(endpoint: &str) -> Result<SocketAddr, anyhow::Error> {\n    let addrs_iter = endpoint.to_socket_addrs()?;\n    for addr in addrs_iter {\n        if is_service_running(addr).await? {\n            return Ok(addr);\n        }\n        eprintln!(\"WARN: {} is not available\", addr);\n    }\n    Err(anyhow!(\"no metasrv running on: {}\", endpoint))\n}\n\npub async fn keys_layout_from_grpc(addr: &str, depth: Option<u32>) -> anyhow::Result<()> {\n    let client = MetaGrpcClient::<DatabendRuntime>::try_create(\n        vec![addr.to_string()],\n        \"root\",\n        \"xxx\",\n        None,\n        None,\n        None,\n        DEFAULT_GRPC_MESSAGE_SIZE,\n    )?;\n\n    let mut grpc_client = client.make_established_client().await?;\n\n    let request = protobuf::KeysLayoutRequest { depth };\n    let response = grpc_client.snapshot_keys_layout(request).await?;\n","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/control/src/keys_layout_from_grpc.rs#L41-L77","documentation":"keys_layout_from_grpc picks the first address in the configured list where a metasrv service is actually running (checked via is_service_running). If none of the addresses respond, it fails with 'no metasrv running on: <endpoint>'.","triggerScenarios":"Running `metactl` subcommands that inspect key layout from a live node when the provided --grpc-api-address points at a host/port where no metasrv listens, or where all candidates in the address list are down.","commonSituations":"Metasrv not started yet, wrong port in --grpc-api-address, firewall/DNS issues, metasrv crashed, or running metactl on a machine that cannot reach the cluster.","solutions":["Start the databend-meta service before running the command","Verify the --grpc-api-address host and port (default 9191) match a running metasrv","Test reachability: curl or nc to the address to confirm the gRPC port is open","Check metasrv logs for startup failure and confirm the address list in the config includes a live node"],"exampleFix":"// before\nmetactl --grpc-api-address 127.0.0.1:9192  # nothing listens here\n// after\nmetactl --grpc-api-address 127.0.0.1:9191  # live metasrv admin API","handlingStrategy":"validation","validationCode":"#!/bin/bash\nADDR=127.0.0.1:9191\nnc -z $(cut -d: -f1 $ADDR) $(cut -d: -f2 $ADDR) || { echo \"metasrv not reachable on $ADDR\"; exit 1; }\n# then run metactl with --grpc-api-address $ADDR","typeGuard":null,"tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"no metasrv running on\") => {\n        eprintln!(\"metasrv is not up at the given address; start databend-meta and retry\");\n    }\n    other => other?,\n}","preventionTips":["Check metasrv health (admin API) before metactl commands","Confirm the gRPC/admin port (default 9191) in configs","Ensure network/firewall allows the metactl host to reach metasrv","Add a readiness wait loop in automation before invoking metactl"],"tags":["network","grpc","metasrv","availability"],"backgroundTag":"connection-refused","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}