risingwavelabs/risingwave · error · HummockError

SstableUpload error

Error message

SstableUpload error: {0}

What it means

HummockError::SstableUploadError (src/storage/src/hummock/error.rs:73-74, constructor sstable_upload_error at error.rs:173-175) wraps failures when uploading newly built SSTable data files and metadata to the object store and/or committing them to meta. It is thrown from the SST upload path shared by streaming compute nodes and the compactor.

Solutions

1. Check object store connectivity and credentials (endpoint, access keys, bucket) from the failing node
2. Retry the flush/checkpoint; uploads are retried per epoch and succeed once storage recovers
3. Inspect the object store service for rate limiting or outages (4xx/5xx in object-store logs)
4. If errors persist on every upload, verify version-compatible object-store configuration in `risedev`/deployment config

Example fix

// before: hard fail on transient object-store blip during upload
let result = upload_sst(&object_store, &sst).map_err(HummockError::sstable_upload_error)?;
// after: retry transient upload failures
let result = retry::retry_async(retry::RETRY_INTERVAL, |_, _| async {
    match upload_sst(&object_store, &sst).await {
        Ok(r) => Ok(r),
        Err(e) if e.is_object_error() => Err(e.into()),
        Err(e) => Err(HummockError::sstable_upload_error(e)),
    }
}).await?;

// before upload-heavy operations, verify object store reachability
let probe = object_store
    .upload("__healthcheck__", Bytes::from_static(b"ok"))
    .await;
assert!(probe.is_ok(), "object store unreachable; sstable uploads will fail");

fn is_sstable_upload_err(e: &HummockError) -> bool {
    e.to_report_string().starts_with("SstableUpload error:")
}

match sync_uploading_sstables().await {
    Err(e) if is_sstable_upload_err(&e) => {
        // uploads are retried per epoch; back off and retry once storage recovers
        warn!(error = %e.report(), "sst upload failed; will retry on next epoch sync");
        retry::retry_async(retry::RETRY_INTERVAL, |_, _| Ok(())).await?;
        sync_uploading_sstables().await
    }
    other => other,
}

When it happens

Trigger: flush_local / sync_uploading_sstables calls fail while PUT-ing SST data to S3/GCS/minio or while the SST meta write to meta fails; the uploader task in HummockStorageEventLoop encounters an object-store error during epoch sync.

Common situations: Object store credentials expired or bucket missing/unreachable; network partition between compute node and object storage; disk/network timeouts during large compaction output upload; S3 rate limits.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• ObjectStore failed with IO error
• Read backup error
• s3 error
• Storage error
• Storage error

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/ba59cb1e4b7c723e. Report an issue: GitHub.

Appendix: source

Thrown at src/storage/src/hummock/error.rs:73

    #[error(
        "Committed epoch mismatch: table {table_id}, committed_epoch {committed_epoch}, read_epoch {read_epoch}"
    )]
    CommittedEpochMismatch {
        table_id: TableId,
        committed_epoch: u64,
        read_epoch: u64,
    },
    #[error("Barrier read is unavailable for now. Likely the cluster is recovering")]
    ReadCurrentEpoch,
    #[error("CompactionExecutor error: {0}")]
    CompactionExecutor(String),
    #[error("FileCache error: {0}")]
    FileCache(String),
    #[error("SstObjectIdTracker error: {0}")]
    SstObjectIdTrackerError(String),
    #[error("CompactionGroup error: {0}")]
    CompactionGroupError(String),
    #[error("SstableUpload error: {0}")]
    SstableUploadError(String),
    #[error("Read backup error: {0}")]
    ReadBackupError(String),
    #[error("Foyer error: {0}")]
    FoyerError(#[from] foyer::Error),
    #[error("Other error: {0}")]
    Other(String),
}

impl HummockError {
    pub fn invalid_format_version(v: u32) -> HummockError {
        HummockErrorInner::InvalidFormatVersion(v).into()
    }

    pub fn invalid_block() -> HummockError {
        HummockErrorInner::InvalidBlock.into()
    }

View on GitHub (pinned to 6469eb736d)