risingwavelabs/risingwave · error

truncate at a later offset {:?} than the current latest offs

Error message

truncate at a later offset {:?} than the current latest offset {:?}

What it means

KvLogStoreReader::truncate refuses to truncate at an offset later than the reader's `latest_offset` — the reader has not consumed that far, so a later truncation would discard data the reader still needs. `latest_offset` is expected to be Some (it panics with 'should exist before truncation' otherwise), and the error here covers the out-of-range case. This guards the invariant that truncation only moves forward within consumed data.

Source

Thrown at src/stream/src/common/log_store_impl/kv_log_store/reader.rs:519

                    item_epoch
                );
                self.latest_offset = Some(TruncateOffset::Barrier { epoch: item_epoch });
                (
                    item_epoch,
                    LogStoreReadItem::Barrier {
                        is_checkpoint,
                        new_vnode_bitmap: None,
                        is_stop,
                        schema_change,
                    },
                )
            }
        })
    }

    fn truncate(&mut self, offset: TruncateOffset) -> LogStoreResult<()> {
        if offset > self.latest_offset.expect("should exist before truncation") {
            return Err(anyhow!(
                "truncate at a later offset {:?} than the current latest offset {:?}",
                offset,
                self.latest_offset
            ));
        }
        if offset.epoch() >= self.first_write_epoch.expect("should have init") {
            if let Some(truncate_offset) = &self.truncate_offset
                && offset <= *truncate_offset
            {
                return Err(anyhow!(
                    "truncate offset {:?} earlier than prev truncate offset {:?}",
                    offset,
                    truncate_offset
                ));
            }
            self.rx.truncate_buffer(offset);
            self.truncate_offset = Some(offset);
        } else {

View on GitHub (pinned to 6469eb736d)

Solutions

1. Only truncate with offsets derived from this reader's own emitted progress, not the writer's.
2. Ensure the reader has consumed up to the target epoch (call next_item until latest_offset passes it) before truncating.
3. Check for duplicate/overlapping truncate calls after failover; discard stale reader instances.
4. Add a debug log comparing the requested offset to latest_offset to catch ordering bugs.

Example fix

// before: truncating beyond consumed progress
let offset = TruncateOffset { epoch: barrier_epoch, .. };
reader.truncate(offset)?;

// after: truncate only up to the reader's latest consumed offset
let offset = reader.latest_offset.min(requested);
reader.truncate(offset)?;

// only truncate offsets this reader has consumed
if offset > reader.latest_offset {
    // clamp or skip instead of calling truncate
    return Ok(());
}
reader.truncate(offset)?;

fn truncatable(offset: TruncateOffset, latest: Option<TruncateOffset>) -> bool {
    matches!(latest, Some(l) if offset <= l)
}

match reader.truncate(offset) {
    Err(e) if e.to_string().contains("truncate at a later offset") => {
        // reader lags or stale instance; re-derive offset from reader progress
        let off = reader.latest_offset();
        reader.truncate(off)?;
    }
    r => r,
}

When it happens

Trigger: Calling `truncate(offset)` with an offset whose epoch (or position within the epoch) exceeds what the reader has already emitted: e.g. truncating based on a writer-side sealed epoch while the reader lags, or a stale reader instance receiving a new truncate request after failover.

Common situations: Reader lagging behind barrier truncation requests; failover where a new owner truncates an old reader; tests or custom callers passing a TruncateOffset derived from the wrong reader's progress.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• truncation epoch {} should not be larger than current epoch
• Filter can only receive bool array
• Exchange executor should not have children!
• GetChannelDeltaStatsExecutor should have no child!
• Iceberg metadata scan should not have input executors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/de3dd5162bc0d05c. Report an issue: GitHub.