risingwavelabs/risingwave · critical · ObjectError

disk error: {msg}

Error message

disk error: {msg}

What it means

A local-disk object store operation failed; the io::Error is wrapped into ObjectError::Disk with the message 'disk error: {msg}'. This surfaces filesystem-level problems (permissions, missing paths, full disk, device errors) from RisingWave's disk-backed object store.

Source

Thrown at src/object_store/src/object/error.rs:36

use aws_sdk_s3::operation::head_object::HeadObjectError;
use aws_sdk_s3::primitives::ByteStreamError;
use aws_smithy_types::body::SdkBody;
use risingwave_common::error::BoxedError;
use thiserror::Error;
use thiserror_ext::AsReport;
use tokio::sync::oneshot::error::RecvError;

#[derive(Error, thiserror_ext::ReportDebug, thiserror_ext::Box, thiserror_ext::Construct)]
#[thiserror_ext(newtype(name = ObjectError, backtrace))]
pub enum ObjectErrorInner {
    #[error("s3 error: {inner}")]
    S3 {
        // TODO: remove this after switch s3 backend to opendal
        should_retry: bool,
        #[source]
        inner: BoxedError,
    },
    #[error("disk error: {msg}")]
    Disk {
        msg: String,
        #[source]
        inner: io::Error,
    },
    #[error(transparent)]
    Opendal(#[from] opendal::Error),
    #[error(transparent)]
    Mem(#[from] crate::object::mem::Error),
    #[error("Internal error: {0}")]
    #[construct(skip)]
    Internal(String),
    #[cfg(madsim)]
    #[error(transparent)]
    Sim(#[from] crate::object::sim::SimError),

    #[error("Timeout error: {0}")]
    Timeout(String),

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the wrapped #[source] io::Error (errno) — ENOSPC means free disk space; EACCES/EPERM means fix permissions.
2. Verify the data directory path exists and is writable by the RisingWave process user.
3. Check mount/volume status (kubectl get pv,pvc; df -h) and expand or remount the volume if full.
4. If the container filesystem is read-only, mount a writable volume for the data directory.
5. Check hardware/disk health (dmesg) for underlying device errors.

Example fix

// before
# risingwave --state-store hummock+disk:///nonexistent-path
// after
mkdir -p /data/risingwave && chown risingwave:risingwave /data/risingwave
# risingwave --state-store hummock+disk:///data/risingwave
# and monitor: df -h /data  (ENOSPC => expand volume)

// preflight the data directory
let dir = std::path::Path::new(data_dir);
std::fs::create_dir_all(dir)?;
let probe = dir.join(".rw-write-probe");
std::fs::write(&probe, b"ok")?;
std::fs::remove_file(&probe)?;

fn disk_err_kind(e: &ObjectError) -> Option<&std::io::Error> {
    match e { ObjectError::Disk { inner, .. } => Some(inner), _ => None }
}

match op().await {
    Err(e) => match disk_err_kind(&e) {
        Some(io) if io.kind() == std::io::ErrorKind::StorageFull => free_space_and_retry(op).await,
        Some(io) => { log::error!("disk: {io}"); Err(e.into()) }
        None => Err(e.into()),
    },
    Ok(v) => Ok(v),
}

When it happens

Trigger: Reading/writing/removing objects on the disk backend when std::fs or tokio::fs returns io::Error: data directory missing, read-only filesystem, permission denied, ENOSPC (disk full), or device I/O error.

Common situations: Pointing state store at a non-existent or unmounted data directory; running containers as a user without write permission on the volume; Kubernetes PV full or evicted; host disk failure; read-only root filesystem in containers.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• check if version hint exist failed: {}
• Fail to check exist
• (dataset guard acquisition error)
• ObjectStore failed with IO error: {0}
• Storage error: {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/efc234b72c737b4c. Report an issue: GitHub.