{"record":{"id":"9084ef638396b676","repo":"risingwavelabs/risingwave","slug":"object-object-id-is-rejected-from-being-committe","errorCode":null,"errorMessage":"object {object_id} is rejected from being committed since it's below watermark: object timestamp {created_at}, meta node timestamp {now}, retention_sec {retention_sec}, watermark {sst_retention_watermark}","messagePattern":"object (.+?) is rejected from being committed since it's below watermark: object timestamp (.+?), meta node timestamp (.+?), retention_sec (.+?), watermark (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/hummock/manager/context.rs","lineNumber":345,"sourceCode":"            object_timestamps.iter().map(|(k, v)| (*k, *v)),\n        )?;\n        if self.env.opts.gc_history_retention_time_sec != 0 {\n            let ids = object_timestamps.keys().copied().collect_vec();\n            check_gc_history(&self.meta_store_ref().conn, ids).await?;\n        }\n        Ok(())\n    }\n}\n\nfn check_sst_retention(\n    now: u64,\n    retention_sec: u64,\n    sst_infos: impl Iterator<Item = (HummockSstableObjectId, u64)>,\n) -> Result<()> {\n    let sst_retention_watermark = now.saturating_sub(retention_sec);\n    for (object_id, created_at) in sst_infos {\n        if created_at < sst_retention_watermark {\n            return Err(anyhow::anyhow!(\"object {object_id} is rejected from being committed since it's below watermark: object timestamp {created_at}, meta node timestamp {now}, retention_sec {retention_sec}, watermark {sst_retention_watermark}\").into());\n        }\n    }\n    Ok(())\n}\n\nasync fn check_gc_history(\n    db: &DatabaseConnection,\n    object_ids: impl IntoIterator<Item = HummockSstableObjectId>,\n) -> Result<()> {\n    let object_ids = object_ids.into_iter().collect_vec();\n    let mut expired_object_ids = Vec::new();\n    for object_ids in object_ids.chunks(GC_HISTORY_QUERY_BATCH_SIZE) {\n        expired_object_ids.extend(\n            hummock_gc_history::Entity::find()\n                .filter(hummock_gc_history::Column::ObjectId.is_in(object_ids.iter().copied()))\n                .all(db)\n                .await?,\n        );","sourceCodeStart":327,"sourceCodeEnd":363,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/hummock/manager/context.rs#L327-L363","documentation":"During commit-time sanity checks, the meta node rejects SSTable objects whose creation timestamp is older than the retention watermark (`now - retention_sec`). This means an object waited so long between creation and being committed to the meta version that its commit would race with (or already have lost to) the object-deletion GC that removes objects below the watermark. The check runs in check_sst_retention, invoked by commit_epoch_sanity_check and report_compaction_sanity_check, to prevent committing objects that GC may have already deleted from object storage.","triggerScenarios":"Calling commit_epochs or report_compaction with SSTs whose created_at timestamp is >= retention_sec behind the meta node's current clock. Occurs when SST creation is severely delayed (huge compactions, backpressure), when retention_sec is very small, or when there is clock skew between compute/compactor nodes and the meta node.","commonSituations":"Under-provisioned meta node with large clock lag; operators lowering hummock retention_sec in config below realistic commit latency; long stalls between SST upload and epoch commit (e.g. checkpoint stuck or slow compaction task reporting); multi-node deployments with unsynchronized NTP clocks.","solutions":["Increase the hummock retention_sec config so the watermark accounts for worst-case commit latency.","Sync clocks across all nodes via NTP/chrony and check for meta node clock skew.","Investigate why commits/compaction reports are delayed (check compactor and meta node metrics, GC pressure, large compaction tasks).","Retry the commit after the transient backlog clears; the rejected objects will typically be re-uploaded or already tracked."],"exampleFix":"// before: hummock config with aggressive retention\n[hummock]\nretention_sec = 3600\n\n// after: allow headroom for slow compaction reporting\n[hummock]\nretention_sec = 86400","handlingStrategy":"retry","validationCode":"// before committing, verify SST freshness against retention\nlet watermark = now.saturating_sub(retention_sec);\nif sst_infos.iter().any(|(_, created_at)| *created_at < watermark) {\n    // warn operator: increase retention_sec or investigate commit delay\n}","typeGuard":null,"tryCatchPattern":"// match on the error and distinguish below-watermark rejections\nmatch meta.commit_epochs(epoch, ssts).await {\n    Err(e) if e.to_string().contains(\"below watermark\") => {\n        // back off and retry; consider raising retention_sec\n    }\n    Err(e) => return Err(e),\n    Ok(v) => Ok(v),\n}","preventionTips":["Set retention_sec generously above worst-case commit/report latency.","Run NTP/chrony on all compute, compactor and meta nodes.","Alert on compaction task age and epoch commit lag before objects age out.","Monitor meta node clock drift in production dashboards."],"tags":["hummock","object-storage","gc","clock-skew","risingwave-meta"],"backgroundTag":"value-out-of-range","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}