risingwavelabs/risingwave · error · Error

compactor {0} is disconnected

Error message

compactor {0} is disconnected

What it means

hummock::Error::CompactorUnreachable indicates that the meta node lost contact with a compactor worker identified by HummockContextId. The compaction scheduler detects that the compactor node (gRPC connection) is gone or unresponsive and marks the context as disconnected, failing pending compaction tasks assigned to it.

Source

Thrown at src/meta/src/hummock/error.rs:42

pub type Result<T> = std::result::Result<T, Error>;

#[derive(Error, Debug)]
pub enum Error {
    #[error("invalid hummock context {0}")]
    InvalidContext(HummockContextId),
    #[error("failed to access meta store")]
    MetaStore(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error(transparent)]
    ObjectStore(
        #[from]
        #[backtrace]
        ObjectError,
    ),
    #[error("compactor {0} is disconnected")]
    CompactorUnreachable(HummockContextId),
    #[error("compaction group error: {0}")]
    CompactionGroup(String),
    #[error("SST {0} is invalid")]
    InvalidSst(HummockSstableObjectId),
    #[error("invalid manual compaction option: {0}")]
    InvalidManualCompactionOption(String),
    #[error("invalid epoch range: {start_epoch}..={end_epoch}")]
    InvalidEpochRange { start_epoch: u64, end_epoch: u64 },
    #[error("time-travel version expired: table {table_id}, epoch {epoch}")]
    TimeTravelVersionExpired { table_id: TableId, epoch: u64 },
    #[error("time travel")]
    TimeTravel(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error(transparent)]

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify compactor pods/processes are running and reconnecting to the meta node
2. Restart the compactor for the given context id and let it re-register
3. Wait for the meta node to reschedule the interrupted compaction tasks to healthy compactors
4. Check network connectivity/MTU issues between meta and compactor nodes
5. Inspect meta logs for when the compactor's heartbeat/connection dropped

Example fix

// before
// assuming task stays assigned to the dead compactor
schedule_task(task, compactor.context_id())?;
// after
if let Err(hummock::Error::CompactorUnreachable(ctx)) = schedule_task(task, compactor.context_id()) {
    tracing::warn!("compactor {ctx} unreachable, rescheduling");
    schedule_task(task, pick_healthy_compactor()?)?;
}

// Check compactor liveness before assigning a task
let alive = health_check(compactor_addr).await.is_ok();
if !alive { reschedule_to_healthy_compactor()?; }

fn is_compactor_unreachable(e: &hummock::Error) -> Option<HummockContextId> {
    match e { hummock::Error::CompactorUnreachable(id) => Some(*id), _ => None }
}

match schedule_result {
    Err(hummock::Error::CompactorUnreachable(ctx)) => {
        tracing::warn!("compactor {ctx} unreachable; rescheduling task");
        scheduler.reassign(task)?
    }
    other => other?,
}

When it happens

Trigger: A compactor process crashed or was killed mid-task; network partition between meta and compactor; compactor gRPC channel closed causing `cancel_task`/task-tracking code to mark the context unreachable; compactor deregistration while tasks are in flight.

Common situations: Compactor OOM-killed or terminated by an orchestrator (k8s pod eviction) during heavy compaction; rolling upgrades removing old compactor nodes; unstable network in self-hosted deployments.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• compaction group error: {0}
• invalid manual compaction option: {0}
• trigger_manual_compaction No compactor is available. compact
• Failed to get compaction task for compaction_group {}
• trigger_manual_compaction No compaction_task is available. c

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/10c119ccb72dc553. Report an issue: GitHub.