{"record":{"id":"88d45315cf023b94","repo":"quickwit-oss/quickwit","slug":"searcher-use-metastore-read-replica-cannot-be-en","errorCode":null,"errorMessage":"`searcher.use_metastore_read_replica` cannot be enabled on a node running the `metastore` service","messagePattern":"`searcher\\.use_metastore_read_replica` cannot be enabled on a node running the `metastore` service","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-config/src/node_config/serialize.rs","lineNumber":447,"sourceCode":"fn validate_metastore_read_replica(node_config: &NodeConfig) -> anyhow::Result<()> {\n    let read_replica_enabled =\n        node_config.is_service_enabled(QuickwitService::MetastoreReadReplica);\n    if read_replica_enabled {\n        ensure!(\n            node_config.enabled_services.len() == 1,\n            \"the `metastore_read_replica` service must run standalone and cannot be combined with \\\n             any other service\"\n        );\n        ensure!(\n            node_config.metastore_read_replica_uri.is_some(),\n            \"the `metastore_read_replica` service requires `metastore_read_replica_uri` to be set\"\n        );\n    }\n    let searcher_uses_read_replica = node_config.is_service_enabled(QuickwitService::Searcher)\n        && node_config.searcher_config.use_metastore_read_replica;\n    // Avoid a deadlock where the searcher waits for a READY read replica, while the read\n    // replica waits for this primary metastore to become READY.\n    ensure!(\n        !(searcher_uses_read_replica && node_config.is_service_enabled(QuickwitService::Metastore)),\n        \"`searcher.use_metastore_read_replica` cannot be enabled on a node running the \\\n         `metastore` service\"\n    );\n    Ok(())\n}\n\n/// A list of all the known disk budgets\n///\n/// External disk usage and unbounded disk usages, e.g the indexing workbench\n/// (indexing/) and the delete task workbench (delete_task_service/) are not included.\n#[derive(Default, Debug)]\nstruct ExpectedDiskUsage {\n    // indexer / ingester\n    split_store_max_num_bytes: Option<ByteSize>,\n    max_queue_disk_usage: Option<ByteSize>,\n    // searcher\n    split_cache: Option<ByteSize>,","sourceCodeStart":429,"sourceCodeEnd":465,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-config/src/node_config/serialize.rs#L429-L465","documentation":"If a node runs both the primary `metastore` service and a searcher configured with `use_metastore_read_replica: true`, a deadlock occurs: the searcher waits for a READY read replica while the read replica waits for the co-hosted primary metastore to become READY. `validate_metastore_read_replica` detects this combination and rejects the config before startup.","triggerScenarios":"A node whose `services` includes `metastore` (and typically `searcher`) with `searcher.use_metastore_read_replica: true` set in the searcher config; validate fails during config load.","commonSituations":"Single-node setups where all services are enabled by default and an operator flips `use_metastore_read_replica` on for search scaling; copied searcher config snippets pasted into an all-in-one node config.","solutions":["Set `searcher.use_metastore_read_replica: false` on nodes running the primary metastore service.","Or remove the `metastore` service from this node and run the primary metastore elsewhere, keeping the read-replica-backed searcher.","Deploy the searcher that uses the read replica as a standalone node pointed at the replica."],"exampleFix":"// before (quickwit.yaml)\nservices: [searcher, metastore]\nsearcher:\n  use_metastore_read_replica: true\n\n// after (quickwit.yaml)\nservices: [searcher, metastore]\nsearcher:\n  use_metastore_read_replica: false","handlingStrategy":"validation","validationCode":"if config.is_service_enabled(QuickwitService::Metastore)\n    && config.searcher_config.use_metastore_read_replica {\n    return Err(\"disable use_metastore_read_replica on nodes running the metastore service\");\n}","typeGuard":"fn searcher_replica_allowed(cfg: &NodeConfig) -> bool {\n    !(cfg.is_service_enabled(QuickwitService::Metastore) && cfg.searcher_config.use_metastore_read_replica)\n}","tryCatchPattern":"if let Err(e) = load_node_config(path) {\n    if e.to_string().contains(\"use_metastore_read_replica\") {\n        eprintln!(\"Use read-replica-backed searchers only on nodes without the metastore service\");\n    }\n    return Err(e);\n}","preventionTips":["Keep use_metastore_read_replica=false in base configs; enable it only in dedicated searcher roles.","Separate primary-metastore nodes from replica-backed searcher nodes in deployment topology docs.","Add a topology test that loads each role's config in CI."],"tags":["config","validation","metastore","searcher","deadlock"],"backgroundTag":"conflicting-config-options","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}