risingwavelabs/risingwave · error · SinkError::LanceDb

failed to read Lance transaction history

Error message

failed to read Lance transaction history

What it means

`is_epoch_committed` walks backward through the Lance dataset's version history (via `read_transaction` and `checkout_version(version - 1)`) looking for a RisingWave epoch/sink-id transaction property. Any error from reading transactions or checking out an older version — other than the expected `DatasetNotFound` that signals truncated history — is wrapped with context 'failed to read Lance transaction history'. Missing versions are treated as 'history fully inspected', but other failures (I/O, auth, corruption) propagate.

Source

Thrown at src/connector/src/sink/lancedb.rs:606

                    .context("invalid RisingWave epoch in Lance transaction history")
                    .map_err(SinkError::LanceDb)?;
                return Ok(committed_epoch >= target_epoch);
            }

            let version = dataset.version().version;
            if version <= 1 {
                return Ok(false);
            }

            dataset = match dataset.checkout_version(version - 1).await {
                Ok(dataset) => dataset,
                // Lance cleanup removes a contiguous prefix of old versions. Reaching a
                // missing previous version therefore means that all retained history has
                // already been inspected.
                Err(lance::Error::DatasetNotFound { .. }) => return Ok(false),
                Err(error) => {
                    return Err(SinkError::LanceDb(
                        anyhow!(error).context("failed to read Lance transaction history"),
                    ));
                }
            };
        }
    }

    async fn commit_fragments(
        &mut self,
        epoch: u64,
        fragments: Vec<Fragment>,
        transaction_properties: Option<HashMap<String, String>>,
    ) -> Result<()> {
        if fragments.is_empty() {
            tracing::debug!("No fragments to commit in epoch {epoch}, skipping.");
            return Ok(());
        }

        // Open the table to get the underlying lance Dataset.

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the wrapped inner error (auth vs I/O vs corruption) in the full error chain and fix that root cause.
2. Verify storage credentials and network access to the dataset's object store from the committer node.
3. Confirm the dataset's _versions and _manifest directories are intact and not being modified by external cleanup jobs.
4. Retry the commit after transient storage failures — the epoch dedup check is idempotent and safe to re-run.

// Retry transient failures of the history walk; treat only DatasetNotFound as end-of-history
match dataset.checkout_version(version - 1).await {
    Ok(d) => { dataset = d; }
    Err(lance::Error::DatasetNotFound { .. }) => return Ok(false),
    Err(e) if is_transient(&e) => {
        tokio::time::sleep(BACKOFF).await;
        continue; // retry the same version
    }
    Err(e) => return Err(SinkError::LanceDb(anyhow!(e).context("failed to read Lance transaction history"))),
}

When it happens

Trigger: During commit_fragments dedup check: `dataset.read_transaction()` fails on the current version, or `checkout_version(version - 1)` fails with an error other than DatasetNotFound — e.g. manifest I/O error, object-store auth failure, network error, or corrupted version/manifest files.

Common situations: Object-store connectivity issues during commit; expired cloud credentials mid-run; externally deleted _versions or _manifest files; concurrent Lance cleanup compaction racing with the history walk.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• (dataset guard acquisition error)
• Invalid commit metadata: empty payload
• Invalid commit metadata: missing snapshot_id
• error acquiring lance dataset guard: {e}
• only append-only LanceDB sink is supported

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/f82b76c94948b165. Report an issue: GitHub.