{"record":{"id":"6d1527cd98427eff","repo":"lancedb/lancedb","slug":"checkpointlsm-the-owning-node-kept-losing-its-cla","errorCode":null,"errorMessage":"checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum number of times","messagePattern":"checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum number of times","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"java/lancedb-core/src/main/java/com/lancedb/LanceDbTableLsm.java","lineNumber":233,"sourceCode":"        backoff(reissue);\n        continue;\n      }\n      if (!stats.value.isPresent()) {\n        // Not WAL-backed; flushLsm would have errored first but for a race.\n        return;\n      }\n\n      Map<String, Long> targets = newestGenerations(stats.value.get());\n      if (targets.isEmpty()) {\n        return;\n      }\n\n      if (drainToTargets(targets)) {\n        return;\n      }\n      backoff(reissue);\n    }\n    throw new IllegalStateException(\n        \"checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum \"\n            + \"number of times\");\n  }\n\n  /**\n   * Trigger and poll until no bucket holds a generation at or below its target.\n   *\n   * @return true when the drain finished, false when the table needs re-claiming from flush.\n   */\n  private boolean drainToTargets(Map<String, Long> targets) {\n    while (true) {\n      Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));\n      if (stats.lostClaim) {\n        return false;\n      }\n      if (!stats.value.isPresent()) {\n        return true;\n      }","sourceCodeStart":215,"sourceCodeEnd":251,"githubUrl":"https://github.com/lancedb/lancedb/blob/c7b051aff7039333a3f61b79217246c27676806a/java/lancedb-core/src/main/java/com/lancedb/LanceDbTableLsm.java#L215-L251","documentation":"checkpointLsm drives an LSM checkpoint to completion by polling buckets against their target generations. The node holding the checkpoint claim must periodically re-issue the checkpoint from flush; if it keeps losing the claim and exhausts the maximum number of re-issue attempts, the library gives up and throws this IllegalStateException.","triggerScenarios":"Calling checkpointLsm (directly or via flush) on a table while another node/instance repeatedly wins and steals the checkpoint ownership claim, so the local node's re-issued attempts exceed the retry cap.","commonSituations":"Multiple writers/checkpointers racing on the same table (multi-process or multi-node deployments), a slow or paused owner that never finishes before its lease expires, or clocks/leases misconfigured so claims expire too quickly under load.","solutions":["Ensure only one writer/checkpointer is active per table, or use a coordination mechanism so claims are not contended","Increase the re-issue/claim retry limits or lease duration in the writer configuration so transient contention is tolerated","Check for stuck or crashed owner processes holding stale claims and clear them","Retry the checkpoint operation later once concurrency on the table has subsided"],"exampleFix":"// before\n// multiple app replicas each calling table.checkpointLsm() concurrently\n// after\n// elect a single checkpointer (e.g. via lock) before checkpointing\nif (acquireCheckpointLock(table)) {\n  try {\n    table.checkpointLsm();\n  } finally {\n    releaseCheckpointLock(table);\n  }\n}","handlingStrategy":"retry","validationCode":"// Check claim contention before checkpointing\n// e.g. verify this process is the designated checkpointer for the table\nboolean isOwner = claimService.tryAcquire(tableIdentifier);\nif (!isOwner) throw new SkipCheckpointException(\"another node owns the claim\");","typeGuard":null,"tryCatchPattern":"try {\n  table.checkpointLsm();\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"kept losing its claim\")) {\n    // back off and retry later with fewer concurrent writers\n    scheduler.schedule(this::checkpoint, backoffSeconds, TimeUnit.SECONDS);\n  } else throw e;\n}","preventionTips":["Run a single checkpointer per table or use leader election","Size lease/retry limits to exceed expected checkpoint duration","Monitor claim-loss events in logs to detect contention early"],"tags":["java","lsm","checkpoint","concurrency"],"backgroundTag":"invalid-state-transition","analyzedSha":"c7b051aff7039333a3f61b79217246c27676806a","analyzedAt":"2026-09-08T23:42:37.579Z","contentChangedAt":"2026-09-08T23:42:37.579Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}