{"record":{"id":"10c119ccb72dc553","repo":"risingwavelabs/risingwave","slug":"compactor-0-is-disconnected","errorCode":null,"errorMessage":"compactor {0} is disconnected","messagePattern":"compactor (.+?) is disconnected","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"src/meta/src/hummock/error.rs","lineNumber":42,"sourceCode":"pub type Result<T> = std::result::Result<T, Error>;\n\n#[derive(Error, Debug)]\npub enum Error {\n    #[error(\"invalid hummock context {0}\")]\n    InvalidContext(HummockContextId),\n    #[error(\"failed to access meta store\")]\n    MetaStore(\n        #[source]\n        #[backtrace]\n        anyhow::Error,\n    ),\n    #[error(transparent)]\n    ObjectStore(\n        #[from]\n        #[backtrace]\n        ObjectError,\n    ),\n    #[error(\"compactor {0} is disconnected\")]\n    CompactorUnreachable(HummockContextId),\n    #[error(\"compaction group error: {0}\")]\n    CompactionGroup(String),\n    #[error(\"SST {0} is invalid\")]\n    InvalidSst(HummockSstableObjectId),\n    #[error(\"invalid manual compaction option: {0}\")]\n    InvalidManualCompactionOption(String),\n    #[error(\"invalid epoch range: {start_epoch}..={end_epoch}\")]\n    InvalidEpochRange { start_epoch: u64, end_epoch: u64 },\n    #[error(\"time-travel version expired: table {table_id}, epoch {epoch}\")]\n    TimeTravelVersionExpired { table_id: TableId, epoch: u64 },\n    #[error(\"time travel\")]\n    TimeTravel(\n        #[source]\n        #[backtrace]\n        anyhow::Error,\n    ),\n    #[error(transparent)]","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/hummock/error.rs#L24-L60","documentation":"hummock::Error::CompactorUnreachable indicates that the meta node lost contact with a compactor worker identified by HummockContextId. The compaction scheduler detects that the compactor node (gRPC connection) is gone or unresponsive and marks the context as disconnected, failing pending compaction tasks assigned to it.","triggerScenarios":"A compactor process crashed or was killed mid-task; network partition between meta and compactor; compactor gRPC channel closed causing `cancel_task`/task-tracking code to mark the context unreachable; compactor deregistration while tasks are in flight.","commonSituations":"Compactor OOM-killed or terminated by an orchestrator (k8s pod eviction) during heavy compaction; rolling upgrades removing old compactor nodes; unstable network in self-hosted deployments.","solutions":["Verify compactor pods/processes are running and reconnecting to the meta node","Restart the compactor for the given context id and let it re-register","Wait for the meta node to reschedule the interrupted compaction tasks to healthy compactors","Check network connectivity/MTU issues between meta and compactor nodes","Inspect meta logs for when the compactor's heartbeat/connection dropped"],"exampleFix":"// before\n// assuming task stays assigned to the dead compactor\nschedule_task(task, compactor.context_id())?;\n// after\nif let Err(hummock::Error::CompactorUnreachable(ctx)) = schedule_task(task, compactor.context_id()) {\n    tracing::warn!(\"compactor {ctx} unreachable, rescheduling\");\n    schedule_task(task, pick_healthy_compactor()?)?;\n}","handlingStrategy":"retry","validationCode":"// Check compactor liveness before assigning a task\nlet alive = health_check(compactor_addr).await.is_ok();\nif !alive { reschedule_to_healthy_compactor()?; }","typeGuard":"fn is_compactor_unreachable(e: &hummock::Error) -> Option<HummockContextId> {\n    match e { hummock::Error::CompactorUnreachable(id) => Some(*id), _ => None }\n}","tryCatchPattern":"match schedule_result {\n    Err(hummock::Error::CompactorUnreachable(ctx)) => {\n        tracing::warn!(\"compactor {ctx} unreachable; rescheduling task\");\n        scheduler.reassign(task)?\n    }\n    other => other?,\n}","preventionTips":["Run compactors under a supervisor/orchestrator with automatic restarts","Keep compactor resource limits sized to avoid OOM kills","Monitor compactor heartbeats and alert on drops","Avoid draining compactors while compaction tasks are mid-flight"],"tags":["compaction","hummock","grpc","connectivity"],"backgroundTag":"connection-refused","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}