object {object_id} is rejected from being committed since it

Error message

object {object_id} is rejected from being committed since it's below watermark: object timestamp {created_at}, meta node timestamp {now}, retention_sec {retention_sec}, watermark {sst_retention_watermark}

What it means

During commit-time sanity checks, the meta node rejects SSTable objects whose creation timestamp is older than the retention watermark (`now - retention_sec`). This means an object waited so long between creation and being committed to the meta version that its commit would race with (or already have lost to) the object-deletion GC that removes objects below the watermark. The check runs in check_sst_retention, invoked by commit_epoch_sanity_check and report_compaction_sanity_check, to prevent committing objects that GC may have already deleted from object storage.

Source

Thrown at src/meta/src/hummock/manager/context.rs:345

            object_timestamps.iter().map(|(k, v)| (*k, *v)),
        )?;
        if self.env.opts.gc_history_retention_time_sec != 0 {
            let ids = object_timestamps.keys().copied().collect_vec();
            check_gc_history(&self.meta_store_ref().conn, ids).await?;
        }
        Ok(())
    }
}

fn check_sst_retention(
    now: u64,
    retention_sec: u64,
    sst_infos: impl Iterator<Item = (HummockSstableObjectId, u64)>,
) -> Result<()> {
    let sst_retention_watermark = now.saturating_sub(retention_sec);
    for (object_id, created_at) in sst_infos {
        if created_at < sst_retention_watermark {
            return Err(anyhow::anyhow!("object {object_id} is rejected from being committed since it's below watermark: object timestamp {created_at}, meta node timestamp {now}, retention_sec {retention_sec}, watermark {sst_retention_watermark}").into());
        }
    }
    Ok(())
}

async fn check_gc_history(
    db: &DatabaseConnection,
    object_ids: impl IntoIterator<Item = HummockSstableObjectId>,
) -> Result<()> {
    let object_ids = object_ids.into_iter().collect_vec();
    let mut expired_object_ids = Vec::new();
    for object_ids in object_ids.chunks(GC_HISTORY_QUERY_BATCH_SIZE) {
        expired_object_ids.extend(
            hummock_gc_history::Entity::find()
                .filter(hummock_gc_history::Column::ObjectId.is_in(object_ids.iter().copied()))
                .all(db)
                .await?,
        );

View on GitHub (pinned to 6469eb736d)

Solutions

Increase the hummock retention_sec config so the watermark accounts for worst-case commit latency.
Sync clocks across all nodes via NTP/chrony and check for meta node clock skew.
Investigate why commits/compaction reports are delayed (check compactor and meta node metrics, GC pressure, large compaction tasks).
Retry the commit after the transient backlog clears; the rejected objects will typically be re-uploaded or already tracked.

Example fix

// before: hummock config with aggressive retention
[hummock]
retention_sec = 3600

// after: allow headroom for slow compaction reporting
[hummock]
retention_sec = 86400

Defensive patterns

Strategy: retry

Validate before calling

// before committing, verify SST freshness against retention
let watermark = now.saturating_sub(retention_sec);
if sst_infos.iter().any(|(_, created_at)| *created_at < watermark) {
    // warn operator: increase retention_sec or investigate commit delay
}

Try / catch

// match on the error and distinguish below-watermark rejections
match meta.commit_epochs(epoch, ssts).await {
    Err(e) if e.to_string().contains("below watermark") => {
        // back off and retry; consider raising retention_sec
    }
    Err(e) => return Err(e),
    Ok(v) => Ok(v),
}

Prevention

Set retention_sec generously above worst-case commit/report latency.
Run NTP/chrony on all compute, compactor and meta nodes.
Alert on compaction task age and epoch commit lag before objects age out.
Monitor meta node clock drift in production dashboards.

When it happens

Trigger: Calling commit_epochs or report_compaction with SSTs whose created_at timestamp is >= retention_sec behind the meta node's current clock. Occurs when SST creation is severely delayed (huge compactions, backpressure), when retention_sec is very small, or when there is clock skew between compute/compactor nodes and the meta node.

Common situations: Under-provisioned meta node with large clock lag; operators lowering hummock retention_sec in config below realistic commit latency; long stalls between SST upload and epoch commit (e.g. checkpoint stuck or slow compaction task reporting); multi-node deployments with unsynchronized NTP clocks.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/9084ef638396b676. Report an issue: GitHub.