risingwavelabs/risingwave · error · BatchError

Failed to spill out to disk

Error message

Failed to spill out to disk

What it means

BatchError::Spill wraps an opendal::Error emitted when the batch engine fails to spill operator data out to disk under memory pressure. Spill uses an OpenDAL operator (local disk or object store) as the backend, and any I/O or backend error surfaced by OpenDAL is converted into this error, aborting the query.

Source

Thrown at src/batch/src/error.rs:146

        Arc<Self>,
    ),

    #[error("Empty workers found")]
    EmptyWorkerNodes,

    #[error("Serving vnode mapping not found for fragment {0}")]
    ServingVnodeMappingNotFound(FragmentId),

    #[error("Streaming vnode mapping has not been initialized")]
    StreamingVnodeMappingNotInitialized,

    #[error("Streaming vnode mapping not found for fragment {0}")]
    StreamingVnodeMappingNotFound(FragmentId),

    #[error("Not enough memory to run this query, batch memory limit is {0} bytes")]
    OutOfMemory(u64),

    #[error("Failed to spill out to disk")]
    Spill(
        #[from]
        #[backtrace]
        opendal::Error,
    ),

    #[error("Failed to execute time travel query")]
    TimeTravel(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
}

// Serialize/deserialize error.
impl From<memcomparable::Error> for BatchError {
    fn from(m: memcomparable::Error) -> Self {
        Self::Serde(m.into())

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check free space and permissions on the spill directory (batch_spill_base_dir, default under the data/tmp path).
2. Verify the SpillBackend configuration (disk vs object store) including endpoint, bucket, and credentials.
3. Free disk space or enlarge the spill volume; clean stale spill files.
4. Retry the query with a higher memory limit to avoid spilling entirely.

Example fix

# before
[system]
... spill backend pointing to a full tmpfs mount ...

# after
# point spill base dir at a persistent volume with free space
RUST_MIN_STACK default; set batch spill base dir to /var/lib/risingwave/spill (real disk)

// Before relying on spill, check the spill directory is writable and has space
let meta = fs::metadata(&spill_dir)?;
// and ensure fs::write(&spill_dir.join(".probe"), b"x") succeeds

match result {
    Err(BatchError::Spill(e)) => {
        // inspect opendal error kind: free disk space / fix backend config, then retry
    }
    other => other?,
}

When it happens

Trigger: A query exceeding the batch memory limit triggers spill; the OpenDAL write to the spill backend fails (disk full, permission denied on the spill directory, object store unavailable/misconfigured), or SpillOp::create fails to open the backend.

Common situations: Disk quota or tmpfs too small for spill data, wrong spill backend configuration (invalid object-store endpoint/credentials), read-only filesystem or missing spill directory permissions in containers.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Spill path must be relative, but got {:?}
• {e}
• Failed to close manifest writer: {}
• OpenDAL is required
• AZBLOB_ENDPOINT not found from environment variables

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b69a74d1dce1db1c. Report an issue: GitHub.