{"record":{"id":"5e117d3252f4649c","repo":"quickwit-oss/quickwit","slug":"compactor-is-enabled-but-no-janitor-node-was-found","errorCode":null,"errorMessage":"compactor is enabled but no janitor node was found in the cluster","messagePattern":"compactor is enabled but no janitor node was found in the cluster","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-serve/src/lib.rs","lineNumber":312,"sourceCode":"    }\n    if is_janitor {\n        let planner = CompactionPlanner::new(metastore_client.clone(), cluster.clone());\n        let (mailbox, handle) = universe.spawn_builder().spawn(planner);\n        info!(\"compaction planner actor started on janitor node\");\n        return Ok((\n            Some(CompactionPlannerServiceClient::from_mailbox(mailbox)),\n            Some(handle),\n        ));\n    }\n    // Compactor-only node: connect to the planner on a remote janitor.\n    let balance_channel = balance_channel_for_service(cluster, QuickwitService::Janitor).await;\n    let found = balance_channel\n        .wait_for(COMPACTION_SERVICE_DISCOVERY_TIMEOUT, |connections| {\n            !connections.is_empty()\n        })\n        .await;\n    if !found {\n        bail!(\"compactor is enabled but no janitor node was found in the cluster\")\n    }\n    info!(\"remote compaction planner detected on janitor node\");\n    Ok((\n        Some(CompactionPlannerServiceClient::from_balance_channel(\n            balance_channel,\n            node_config.grpc_config.max_message_size,\n            None,\n        )),\n        None,\n    ))\n}\n\nfn spawn_merge_scheduler_service(\n    universe: &Universe,\n    node_config: &NodeConfig,\n) -> Mailbox<MergeSchedulerService> {\n    let (mailbox, _) = universe.spawn_builder().spawn(MergeSchedulerService::new(\n        node_config.indexer_config.merge_concurrency.get(),","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-serve/src/lib.rs#L294-L330","documentation":"When the compactor role is enabled, Quickwit's server discovers a janitor node hosting the remote CompactionPlannerService via gRPC balance-channel service discovery. If no janitor connection appears within COMPACTION_SERVICE_DISCOVERY_TIMEOUT, serve_quickwit bails with this message instead of running compaction without a planner.","triggerScenarios":"Starting a node with the compactor enabled in the config while (a) no node in the cluster runs the janitor role, (b) the janitor node is down or unreachable via gRPC, or (c) the janitor has not yet registered its gRPC service in Chitchat within the discovery timeout window.","commonSituations":"Misconfigured cluster where roles are split but the janitor role was omitted; janitor deployment crashed or scaled to 0; networking/firewall blocking the gRPC port; single-node setups enabling compactor but not janitor; transient startup ordering where compactor starts before the janitor registers.","solutions":["Enable the janitor role in the cluster config so at least one node advertises it (or run janitor on the same node in single-node setups).","Verify the janitor node is running and its gRPC port is reachable (curl/nc the address, check Chitchat cluster member list).","Wait for/restart the janitor until it registers with the cluster, then retry the compactor start.","If remote compaction is not wanted, disable the compactor role instead of leaving it enabled without a janitor."],"exampleFix":"// before (node_config: compactor enabled, janitor missing)\n[compactor]\nenable = true\n// after (add janitor role to the cluster)\n[janitor]\nenable = true\n[compactor]\nenable = true","handlingStrategy":"validation","validationCode":"// Before starting the compactor, verify a janitor is advertised in the cluster\nconst members = await fetch('http://node:7280/health/livez') &&\n  await fetchClusterMembers(); // e.g. via Chitchat/admin endpoint\nif (!members.some(m => m.roles.includes('janitor'))) {\n  throw new Error('Refusing to enable compactor: no janitor node in cluster');\n}","typeGuard":"function hasJanitor(members) {\n  return Array.isArray(members) &&\n    members.some(m => Array.isArray(m.roles) && m.roles.includes('janitor'));\n}","tryCatchPattern":"try {\n  await startQuickwitNode(config);\n} catch (e) {\n  if (e.message.includes('no janitor node was found')) {\n    console.error('Enable the janitor role on a cluster node or disable the compactor.');\n  }\n  process.exit(1);\n}","preventionTips":["Keep role assignments (compactor requires janitor) in a validated deployment manifest or Helm values.","Health-check janitor gRPC endpoints before rolling out compactor nodes.","In single-node setups, enable janitor alongside compactor.","Monitor Chitchat membership so janitor loss is alerted before compactor restarts fail."],"tags":["grpc","service-discovery","cluster","compaction","janitor"],"backgroundTag":"resource-not-found","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}