{"record":{"id":"ba59cb1e4b7c723e","repo":"risingwavelabs/risingwave","slug":"sstableupload-error-0","errorCode":null,"errorMessage":"SstableUpload error: {0}","messagePattern":"SstableUpload error: (.+?)","errorType":"exception","errorClass":"HummockError","httpStatus":null,"severity":"error","filePath":"src/storage/src/hummock/error.rs","lineNumber":73,"sourceCode":"    #[error(\n        \"Committed epoch mismatch: table {table_id}, committed_epoch {committed_epoch}, read_epoch {read_epoch}\"\n    )]\n    CommittedEpochMismatch {\n        table_id: TableId,\n        committed_epoch: u64,\n        read_epoch: u64,\n    },\n    #[error(\"Barrier read is unavailable for now. Likely the cluster is recovering\")]\n    ReadCurrentEpoch,\n    #[error(\"CompactionExecutor error: {0}\")]\n    CompactionExecutor(String),\n    #[error(\"FileCache error: {0}\")]\n    FileCache(String),\n    #[error(\"SstObjectIdTracker error: {0}\")]\n    SstObjectIdTrackerError(String),\n    #[error(\"CompactionGroup error: {0}\")]\n    CompactionGroupError(String),\n    #[error(\"SstableUpload error: {0}\")]\n    SstableUploadError(String),\n    #[error(\"Read backup error: {0}\")]\n    ReadBackupError(String),\n    #[error(\"Foyer error: {0}\")]\n    FoyerError(#[from] foyer::Error),\n    #[error(\"Other error: {0}\")]\n    Other(String),\n}\n\nimpl HummockError {\n    pub fn invalid_format_version(v: u32) -> HummockError {\n        HummockErrorInner::InvalidFormatVersion(v).into()\n    }\n\n    pub fn invalid_block() -> HummockError {\n        HummockErrorInner::InvalidBlock.into()\n    }\n","sourceCodeStart":55,"sourceCodeEnd":91,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/storage/src/hummock/error.rs#L55-L91","documentation":"HummockError::SstableUploadError (src/storage/src/hummock/error.rs:73-74, constructor sstable_upload_error at error.rs:173-175) wraps failures when uploading newly built SSTable data files and metadata to the object store and/or committing them to meta. It is thrown from the SST upload path shared by streaming compute nodes and the compactor.","triggerScenarios":"flush_local / sync_uploading_sstables calls fail while PUT-ing SST data to S3/GCS/minio or while the SST meta write to meta fails; the uploader task in HummockStorageEventLoop encounters an object-store error during epoch sync.","commonSituations":"Object store credentials expired or bucket missing/unreachable; network partition between compute node and object storage; disk/network timeouts during large compaction output upload; S3 rate limits.","solutions":["Check object store connectivity and credentials (endpoint, access keys, bucket) from the failing node","Retry the flush/checkpoint; uploads are retried per epoch and succeed once storage recovers","Inspect the object store service for rate limiting or outages (4xx/5xx in object-store logs)","If errors persist on every upload, verify version-compatible object-store configuration in `risedev`/deployment config"],"exampleFix":"// before: hard fail on transient object-store blip during upload\nlet result = upload_sst(&object_store, &sst).map_err(HummockError::sstable_upload_error)?;\n// after: retry transient upload failures\nlet result = retry::retry_async(retry::RETRY_INTERVAL, |_, _| async {\n    match upload_sst(&object_store, &sst).await {\n        Ok(r) => Ok(r),\n        Err(e) if e.is_object_error() => Err(e.into()),\n        Err(e) => Err(HummockError::sstable_upload_error(e)),\n    }\n}).await?;","handlingStrategy":"retry","validationCode":"// before upload-heavy operations, verify object store reachability\nlet probe = object_store\n    .upload(\"__healthcheck__\", Bytes::from_static(b\"ok\"))\n    .await;\nassert!(probe.is_ok(), \"object store unreachable; sstable uploads will fail\");","typeGuard":"fn is_sstable_upload_err(e: &HummockError) -> bool {\n    e.to_report_string().starts_with(\"SstableUpload error:\")\n}","tryCatchPattern":"match sync_uploading_sstables().await {\n    Err(e) if is_sstable_upload_err(&e) => {\n        // uploads are retried per epoch; back off and retry once storage recovers\n        warn!(error = %e.report(), \"sst upload failed; will retry on next epoch sync\");\n        retry::retry_async(retry::RETRY_INTERVAL, |_, _| Ok(())).await?;\n        sync_uploading_sstables().await\n    }\n    other => other,\n}","preventionTips":["Use long-lived, rotated-on-schedule object store credentials and monitor expiry","Alert on object store 4xx/5xx rates and network RTT from compute/compactor nodes","Size epochs/checkpoints so single uploads stay within object store payload limits","Keep endpoint/bucket config consistent across all nodes in the cluster"],"tags":["storage","hummock","object-store","upload","s3"],"backgroundTag":"file-write-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}