{"record":{"id":"b69a74d1dce1db1c","repo":"risingwavelabs/risingwave","slug":"failed-to-spill-out-to-disk","errorCode":null,"errorMessage":"Failed to spill out to disk","messagePattern":"Failed to spill out to disk","errorType":"error_code","errorClass":"BatchError","httpStatus":null,"severity":"error","filePath":"src/batch/src/error.rs","lineNumber":146,"sourceCode":"        Arc<Self>,\n    ),\n\n    #[error(\"Empty workers found\")]\n    EmptyWorkerNodes,\n\n    #[error(\"Serving vnode mapping not found for fragment {0}\")]\n    ServingVnodeMappingNotFound(FragmentId),\n\n    #[error(\"Streaming vnode mapping has not been initialized\")]\n    StreamingVnodeMappingNotInitialized,\n\n    #[error(\"Streaming vnode mapping not found for fragment {0}\")]\n    StreamingVnodeMappingNotFound(FragmentId),\n\n    #[error(\"Not enough memory to run this query, batch memory limit is {0} bytes\")]\n    OutOfMemory(u64),\n\n    #[error(\"Failed to spill out to disk\")]\n    Spill(\n        #[from]\n        #[backtrace]\n        opendal::Error,\n    ),\n\n    #[error(\"Failed to execute time travel query\")]\n    TimeTravel(\n        #[source]\n        #[backtrace]\n        anyhow::Error,\n    ),\n}\n\n// Serialize/deserialize error.\nimpl From<memcomparable::Error> for BatchError {\n    fn from(m: memcomparable::Error) -> Self {\n        Self::Serde(m.into())","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/batch/src/error.rs#L128-L164","documentation":"BatchError::Spill wraps an opendal::Error emitted when the batch engine fails to spill operator data out to disk under memory pressure. Spill uses an OpenDAL operator (local disk or object store) as the backend, and any I/O or backend error surfaced by OpenDAL is converted into this error, aborting the query.","triggerScenarios":"A query exceeding the batch memory limit triggers spill; the OpenDAL write to the spill backend fails (disk full, permission denied on the spill directory, object store unavailable/misconfigured), or SpillOp::create fails to open the backend.","commonSituations":"Disk quota or tmpfs too small for spill data, wrong spill backend configuration (invalid object-store endpoint/credentials), read-only filesystem or missing spill directory permissions in containers.","solutions":["Check free space and permissions on the spill directory (batch_spill_base_dir, default under the data/tmp path).","Verify the SpillBackend configuration (disk vs object store) including endpoint, bucket, and credentials.","Free disk space or enlarge the spill volume; clean stale spill files.","Retry the query with a higher memory limit to avoid spilling entirely."],"exampleFix":"# before\n[system]\n... spill backend pointing to a full tmpfs mount ...\n\n# after\n# point spill base dir at a persistent volume with free space\nRUST_MIN_STACK default; set batch spill base dir to /var/lib/risingwave/spill (real disk)","handlingStrategy":"try-catch","validationCode":"// Before relying on spill, check the spill directory is writable and has space\nlet meta = fs::metadata(&spill_dir)?;\n// and ensure fs::write(&spill_dir.join(\".probe\"), b\"x\") succeeds","typeGuard":null,"tryCatchPattern":"match result {\n    Err(BatchError::Spill(e)) => {\n        // inspect opendal error kind: free disk space / fix backend config, then retry\n    }\n    other => other?,\n}","preventionTips":["Provision the spill volume with several times the batch memory limit of free space.","Validate spill backend (endpoint/credentials/permissions) in deployment health checks.","Use persistent real-disk storage rather than small tmpfs for spill.","Clean up stale spill files on startup."],"tags":["spill","opendal","disk-io","memory-management"],"backgroundTag":"file-write-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}