risingwavelabs/risingwave · error · HummockError

SstableUpload error: {0}

Error message

SstableUpload error: {0}

What it means

HummockError::SstableUploadError (src/storage/src/hummock/error.rs:73-74, constructor sstable_upload_error at error.rs:173-175) wraps failures when uploading newly built SSTable data files and metadata to the object store and/or committing them to meta. It is thrown from the SST upload path shared by streaming compute nodes and the compactor.

Source

Thrown at src/storage/src/hummock/error.rs:73

    #[error(
        "Committed epoch mismatch: table {table_id}, committed_epoch {committed_epoch}, read_epoch {read_epoch}"
    )]
    CommittedEpochMismatch {
        table_id: TableId,
        committed_epoch: u64,
        read_epoch: u64,
    },
    #[error("Barrier read is unavailable for now. Likely the cluster is recovering")]
    ReadCurrentEpoch,
    #[error("CompactionExecutor error: {0}")]
    CompactionExecutor(String),
    #[error("FileCache error: {0}")]
    FileCache(String),
    #[error("SstObjectIdTracker error: {0}")]
    SstObjectIdTrackerError(String),
    #[error("CompactionGroup error: {0}")]
    CompactionGroupError(String),
    #[error("SstableUpload error: {0}")]
    SstableUploadError(String),
    #[error("Read backup error: {0}")]
    ReadBackupError(String),
    #[error("Foyer error: {0}")]
    FoyerError(#[from] foyer::Error),
    #[error("Other error: {0}")]
    Other(String),
}

impl HummockError {
    pub fn invalid_format_version(v: u32) -> HummockError {
        HummockErrorInner::InvalidFormatVersion(v).into()
    }

    pub fn invalid_block() -> HummockError {
        HummockErrorInner::InvalidBlock.into()
    }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check object store connectivity and credentials (endpoint, access keys, bucket) from the failing node
2. Retry the flush/checkpoint; uploads are retried per epoch and succeed once storage recovers
3. Inspect the object store service for rate limiting or outages (4xx/5xx in object-store logs)
4. If errors persist on every upload, verify version-compatible object-store configuration in `risedev`/deployment config

Example fix

// before: hard fail on transient object-store blip during upload
let result = upload_sst(&object_store, &sst).map_err(HummockError::sstable_upload_error)?;
// after: retry transient upload failures
let result = retry::retry_async(retry::RETRY_INTERVAL, |_, _| async {
    match upload_sst(&object_store, &sst).await {
        Ok(r) => Ok(r),
        Err(e) if e.is_object_error() => Err(e.into()),
        Err(e) => Err(HummockError::sstable_upload_error(e)),
    }
}).await?;

// before upload-heavy operations, verify object store reachability
let probe = object_store
    .upload("__healthcheck__", Bytes::from_static(b"ok"))
    .await;
assert!(probe.is_ok(), "object store unreachable; sstable uploads will fail");

fn is_sstable_upload_err(e: &HummockError) -> bool {
    e.to_report_string().starts_with("SstableUpload error:")
}

match sync_uploading_sstables().await {
    Err(e) if is_sstable_upload_err(&e) => {
        // uploads are retried per epoch; back off and retry once storage recovers
        warn!(error = %e.report(), "sst upload failed; will retry on next epoch sync");
        retry::retry_async(retry::RETRY_INTERVAL, |_, _| Ok(())).await?;
        sync_uploading_sstables().await
    }
    other => other,
}

When it happens

Trigger: flush_local / sync_uploading_sstables calls fail while PUT-ing SST data to S3/GCS/minio or while the SST meta write to meta fails; the uploader task in HummockStorageEventLoop encounters an object-store error during epoch sync.

Common situations: Object store credentials expired or bucket missing/unreachable; network partition between compute node and object storage; disk/network timeouts during large compaction output upload; S3 rate limits.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Storage error: {0}
• Storage error: {0}
• StorageError {error}
• s3 error: {inner}
• ObjectStore failed with IO error: {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/ba59cb1e4b7c723e. Report an issue: GitHub.