{"record":{"id":"f82b76c94948b165","repo":"risingwavelabs/risingwave","slug":"failed-to-read-lance-transaction-history","errorCode":null,"errorMessage":"failed to read Lance transaction history","messagePattern":"failed to read Lance transaction history","errorType":"exception","errorClass":"SinkError::LanceDb","httpStatus":null,"severity":"error","filePath":"src/connector/src/sink/lancedb.rs","lineNumber":606,"sourceCode":"                    .context(\"invalid RisingWave epoch in Lance transaction history\")\n                    .map_err(SinkError::LanceDb)?;\n                return Ok(committed_epoch >= target_epoch);\n            }\n\n            let version = dataset.version().version;\n            if version <= 1 {\n                return Ok(false);\n            }\n\n            dataset = match dataset.checkout_version(version - 1).await {\n                Ok(dataset) => dataset,\n                // Lance cleanup removes a contiguous prefix of old versions. Reaching a\n                // missing previous version therefore means that all retained history has\n                // already been inspected.\n                Err(lance::Error::DatasetNotFound { .. }) => return Ok(false),\n                Err(error) => {\n                    return Err(SinkError::LanceDb(\n                        anyhow!(error).context(\"failed to read Lance transaction history\"),\n                    ));\n                }\n            };\n        }\n    }\n\n    async fn commit_fragments(\n        &mut self,\n        epoch: u64,\n        fragments: Vec<Fragment>,\n        transaction_properties: Option<HashMap<String, String>>,\n    ) -> Result<()> {\n        if fragments.is_empty() {\n            tracing::debug!(\"No fragments to commit in epoch {epoch}, skipping.\");\n            return Ok(());\n        }\n\n        // Open the table to get the underlying lance Dataset.","sourceCodeStart":588,"sourceCodeEnd":624,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/lancedb.rs#L588-L624","documentation":"`is_epoch_committed` walks backward through the Lance dataset's version history (via `read_transaction` and `checkout_version(version - 1)`) looking for a RisingWave epoch/sink-id transaction property. Any error from reading transactions or checking out an older version — other than the expected `DatasetNotFound` that signals truncated history — is wrapped with context 'failed to read Lance transaction history'. Missing versions are treated as 'history fully inspected', but other failures (I/O, auth, corruption) propagate.","triggerScenarios":"During commit_fragments dedup check: `dataset.read_transaction()` fails on the current version, or `checkout_version(version - 1)` fails with an error other than DatasetNotFound — e.g. manifest I/O error, object-store auth failure, network error, or corrupted version/manifest files.","commonSituations":"Object-store connectivity issues during commit; expired cloud credentials mid-run; externally deleted _versions or _manifest files; concurrent Lance cleanup compaction racing with the history walk.","solutions":["Inspect the wrapped inner error (auth vs I/O vs corruption) in the full error chain and fix that root cause.","Verify storage credentials and network access to the dataset's object store from the committer node.","Confirm the dataset's _versions and _manifest directories are intact and not being modified by external cleanup jobs.","Retry the commit after transient storage failures — the epoch dedup check is idempotent and safe to re-run."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Retry transient failures of the history walk; treat only DatasetNotFound as end-of-history\nmatch dataset.checkout_version(version - 1).await {\n    Ok(d) => { dataset = d; }\n    Err(lance::Error::DatasetNotFound { .. }) => return Ok(false),\n    Err(e) if is_transient(&e) => {\n        tokio::time::sleep(BACKOFF).await;\n        continue; // retry the same version\n    }\n    Err(e) => return Err(SinkError::LanceDb(anyhow!(e).context(\"failed to read Lance transaction history\"))),\n}","preventionTips":["Ensure commit-time credentials and network access to the object store are valid and long-lived","Do not run external Lance cleanup/compaction that races with the sink committer","Treat this error as retryable — the epoch dedup check is idempotent","Alert on repeated manifest-read failures as a sign of storage instability"],"tags":["lancedb","rust","storage","transaction-history","commit"],"backgroundTag":"database-query-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}