microsoft/FASTER · error

Unexpected checkpoint phase during recovery

Error message

Unexpected checkpoint phase during recovery [expected: REST, actual: %s]

What it means

Recover() must start from the REST phase; it atomically swaps phase REST -> RECOVER and returns Status::Aborted if the current phase is anything else. This prevents recovery from racing with an in-progress checkpoint, which would read inconsistent store state.

Solutions

1. Ensure no checkpoint is in flight before calling Recover
2. Only call Recover once during session setup and from a single thread
3. Retry Recover after the current checkpoint completes and phase returns to REST
4. Check the returned Status::Aborted and re-attempt later instead of ignoring it

Example fix

// before
f2->Checkpoint();
f2->Recover(token, version, sessions); // may abort: checkpoint in flight
// after
wait_for_checkpoint_completion(f2);
Status s = f2->Recover(token, version, sessions);
if (s != Status::Ok) { /* retry later */ }

// ensure no checkpoint active: app-level flag set around Checkpoint()
if (checkpointing.load()) { wait_until_rest(); }

Status s = f2->Recover(token, v, ids);
if (s == Status::Aborted) { /* checkpoint in flight; retry after it completes */ }

When it happens

Trigger: Calling Recover(token, ...) while a checkpoint is in progress (phase in HOT/COLD_STORE_CHECKPOINT) or while another Recover is already running.

Common situations: Calling Recover concurrently from multiple sessions; issuing recovery right after a Checkpoint() that hasn't finished; application startup code racing a background checkpoint thread.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Unable to set first valid segment to
• Unable to set last valid segment to
• Invalid checkpoint version
• Invalid checksum for checkpoint
• Invalid log commit metadata for ID

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/c8e1893de5734b88. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:698

    }
    log_debug("Moving to cold-store checkpoint phase");
    checkpoint->phase.store(CheckpointPhase::COLD_STORE_CHECKPOINT);

    if ((status = checkpoint->cold_store_status.load()) != StoreCheckpointStatus::IDLE) {
      log_error("Unexpected checkpoint status for COLD store [expected: IDLE, actual: %s]",
                STORE_CHECKPOINT_STATUS_STR[static_cast<int>(status)]);
      assert(false);
    }
    log_debug("Requesting cold store checkpoint...");
    checkpoint->cold_store_status.store(StoreCheckpointStatus::REQUESTED);
  }
}

template<class K, class V, class D, class HHI, class CHI>
inline Status F2Kv<K, V, D, HHI, CHI>::Recover(const Guid& token, uint32_t& version, std::vector<Guid>& session_ids) {
  CheckpointPhase phase = CheckpointPhase::REST;
  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::RECOVER)) {
    log_error("Unexpected checkpoint phase during recovery [expected: REST, actual: %s]",
              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
    return Status::Aborted;
  }

  uint32_t hot_store_version, cold_store_version;
  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {
    log_error("Failed to recover hot store!");
    return Status::Aborted;
  }

  std::vector<Guid> temp_vector;
  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {
    log_error("Failed to recover cold store!");
    return Status::Aborted;
  }

  if (hot_store_version != cold_store_version) {
    log_warn("Version of stores differ [hot: %u, cold %u]", hot_store_version, cold_store_version);

View on GitHub (pinned to 321d872eab)