{"record":{"id":"efc234b72c737b4c","repo":"risingwavelabs/risingwave","slug":"disk-error-msg","errorCode":null,"errorMessage":"disk error: {msg}","messagePattern":"disk error: (.+?)","errorType":"exception","errorClass":"ObjectError","httpStatus":null,"severity":"critical","filePath":"src/object_store/src/object/error.rs","lineNumber":36,"sourceCode":"use aws_sdk_s3::operation::head_object::HeadObjectError;\nuse aws_sdk_s3::primitives::ByteStreamError;\nuse aws_smithy_types::body::SdkBody;\nuse risingwave_common::error::BoxedError;\nuse thiserror::Error;\nuse thiserror_ext::AsReport;\nuse tokio::sync::oneshot::error::RecvError;\n\n#[derive(Error, thiserror_ext::ReportDebug, thiserror_ext::Box, thiserror_ext::Construct)]\n#[thiserror_ext(newtype(name = ObjectError, backtrace))]\npub enum ObjectErrorInner {\n    #[error(\"s3 error: {inner}\")]\n    S3 {\n        // TODO: remove this after switch s3 backend to opendal\n        should_retry: bool,\n        #[source]\n        inner: BoxedError,\n    },\n    #[error(\"disk error: {msg}\")]\n    Disk {\n        msg: String,\n        #[source]\n        inner: io::Error,\n    },\n    #[error(transparent)]\n    Opendal(#[from] opendal::Error),\n    #[error(transparent)]\n    Mem(#[from] crate::object::mem::Error),\n    #[error(\"Internal error: {0}\")]\n    #[construct(skip)]\n    Internal(String),\n    #[cfg(madsim)]\n    #[error(transparent)]\n    Sim(#[from] crate::object::sim::SimError),\n\n    #[error(\"Timeout error: {0}\")]\n    Timeout(String),","sourceCodeStart":18,"sourceCodeEnd":54,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/object_store/src/object/error.rs#L18-L54","documentation":"A local-disk object store operation failed; the io::Error is wrapped into ObjectError::Disk with the message 'disk error: {msg}'. This surfaces filesystem-level problems (permissions, missing paths, full disk, device errors) from RisingWave's disk-backed object store.","triggerScenarios":"Reading/writing/removing objects on the disk backend when std::fs or tokio::fs returns io::Error: data directory missing, read-only filesystem, permission denied, ENOSPC (disk full), or device I/O error.","commonSituations":"Pointing state store at a non-existent or unmounted data directory; running containers as a user without write permission on the volume; Kubernetes PV full or evicted; host disk failure; read-only root filesystem in containers.","solutions":["Read the wrapped #[source] io::Error (errno) — ENOSPC means free disk space; EACCES/EPERM means fix permissions.","Verify the data directory path exists and is writable by the RisingWave process user.","Check mount/volume status (kubectl get pv,pvc; df -h) and expand or remount the volume if full.","If the container filesystem is read-only, mount a writable volume for the data directory.","Check hardware/disk health (dmesg) for underlying device errors."],"exampleFix":"// before\n# risingwave --state-store hummock+disk:///nonexistent-path\n// after\nmkdir -p /data/risingwave && chown risingwave:risingwave /data/risingwave\n# risingwave --state-store hummock+disk:///data/risingwave\n# and monitor: df -h /data  (ENOSPC => expand volume)","handlingStrategy":"validation","validationCode":"// preflight the data directory\nlet dir = std::path::Path::new(data_dir);\nstd::fs::create_dir_all(dir)?;\nlet probe = dir.join(\".rw-write-probe\");\nstd::fs::write(&probe, b\"ok\")?;\nstd::fs::remove_file(&probe)?;","typeGuard":"fn disk_err_kind(e: &ObjectError) -> Option<&std::io::Error> {\n    match e { ObjectError::Disk { inner, .. } => Some(inner), _ => None }\n}","tryCatchPattern":"match op().await {\n    Err(e) => match disk_err_kind(&e) {\n        Some(io) if io.kind() == std::io::ErrorKind::StorageFull => free_space_and_retry(op).await,\n        Some(io) => { log::error!(\"disk: {io}\"); Err(e.into()) }\n        None => Err(e.into()),\n    },\n    Ok(v) => Ok(v),\n}","preventionTips":["Preflight create/write/delete on the data directory at process start.","Run the process as a user owning the data volume; never run on read-only rootfs.","Monitor disk usage and alert before ENOSPC.","Use a dedicated mounted volume for hummock disk state store.","Check dmesg/device health when I/O errors recur."],"tags":["disk","filesystem","object-store","storage","io"],"backgroundTag":"file-write-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}