lancedb/lancedb · error · IllegalStateException

checkpointLsm: the owning node kept losing its claim…

Error message

checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum number of times

What it means

checkpointLsm drives an LSM checkpoint to completion by polling buckets against their target generations. The node holding the checkpoint claim must periodically re-issue the checkpoint from flush; if it keeps losing the claim and exhausts the maximum number of re-issue attempts, the library gives up and throws this IllegalStateException.

Solutions

1. Ensure only one writer/checkpointer is active per table, or use a coordination mechanism so claims are not contended
2. Increase the re-issue/claim retry limits or lease duration in the writer configuration so transient contention is tolerated
3. Check for stuck or crashed owner processes holding stale claims and clear them
4. Retry the checkpoint operation later once concurrency on the table has subsided

Example fix

// before
// multiple app replicas each calling table.checkpointLsm() concurrently
// after
// elect a single checkpointer (e.g. via lock) before checkpointing
if (acquireCheckpointLock(table)) {
  try {
    table.checkpointLsm();
  } finally {
    releaseCheckpointLock(table);
  }
}

// Check claim contention before checkpointing
// e.g. verify this process is the designated checkpointer for the table
boolean isOwner = claimService.tryAcquire(tableIdentifier);
if (!isOwner) throw new SkipCheckpointException("another node owns the claim");

try {
  table.checkpointLsm();
} catch (IllegalStateException e) {
  if (e.getMessage().contains("kept losing its claim")) {
    // back off and retry later with fewer concurrent writers
    scheduler.schedule(this::checkpoint, backoffSeconds, TimeUnit.SECONDS);
  } else throw e;
}

When it happens

Trigger: Calling checkpointLsm (directly or via flush) on a table while another node/instance repeatedly wins and steals the checkpoint ownership claim, so the local node's re-issued attempts exceed the retry cap.

Common situations: Multiple writers/checkpointers racing on the same table (multi-process or multi-node deployments), a slow or paused owner that never finishes before its lease expires, or clocks/leases misconfigured so claims expire too quickly under load.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Interrupted while waiting on the LSM checkpoint
• Column cannot be null or empty
• get_lsm_stats returned an empty response body
• get_lsm_write_spec response has no sharding mode
• Spec cannot be null

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/6d1527cd98427eff. Report an issue: GitHub.

Appendix: source

Thrown at java/lancedb-core/src/main/java/com/lancedb/LanceDbTableLsm.java:233

        backoff(reissue);
        continue;
      }
      if (!stats.value.isPresent()) {
        // Not WAL-backed; flushLsm would have errored first but for a race.
        return;
      }

      Map<String, Long> targets = newestGenerations(stats.value.get());
      if (targets.isEmpty()) {
        return;
      }

      if (drainToTargets(targets)) {
        return;
      }
      backoff(reissue);
    }
    throw new IllegalStateException(
        "checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum "
            + "number of times");
  }

  /**
   * Trigger and poll until no bucket holds a generation at or below its target.
   *
   * @return true when the drain finished, false when the table needs re-claiming from flush.
   */
  private boolean drainToTargets(Map<String, Long> targets) {
    while (true) {
      Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));
      if (stats.lostClaim) {
        return false;
      }
      if (!stats.value.isPresent()) {
        return true;
      }

View on GitHub (pinned to c7b051aff7)