microsoft/FASTER · error

Unexpected checkpoint phase during recovery

Error message

Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]

What it means

At the end of Recover(), the phase must be RECOVER before being swapped back to REST. If the CAS fails, another thread altered the checkpoint phase during recovery (e.g., started a checkpoint concurrently), so recovery returns Status::Aborted even though stores may have loaded.

Solutions

1. Ensure no other thread calls Checkpoint()/Recover() during recovery
2. Call Recover exactly once per session, from a single thread, before starting any checkpoint activity
3. Treat the returned Status::Aborted as a real failure and redo recovery rather than proceeding
4. Re-open the instance to return the state machine to REST before retrying

Example fix

// before
std::thread(recover_fn); std::thread(checkpoint_fn); // both mutate phase
// after
recover_fn(); // complete recovery first
std::thread(checkpoint_fn); // then checkpoint

std::lock_guard<std::mutex> g(recover_mu); // prevents concurrent phase mutation

Status s = f2->Recover(token, v, ids);
if (s == Status::Aborted) { /* reopen or re-run recovery single-threaded */ }

When it happens

Trigger: Calling Checkpoint() (or a second Recover) from another thread while Recover() is executing; background checkpoint logic interleaving with recovery.

Common situations: Startup code that triggers recovery on one thread while a maintenance thread kicks off a checkpoint; duplicated session-setup code paths calling Recover twice.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Unexpected checkpoint phase
• Unexpected checkpoint phase
• Unexpected checkpoint status for COLD store
• Unexpected checkpoint phase during recovery
• Unexpected sealed buffer found

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/60dd732a7fb01b6b. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:721

  uint32_t hot_store_version, cold_store_version;
  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {
    log_error("Failed to recover hot store!");
    return Status::Aborted;
  }

  std::vector<Guid> temp_vector;
  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {
    log_error("Failed to recover cold store!");
    return Status::Aborted;
  }

  if (hot_store_version != cold_store_version) {
    log_warn("Version of stores differ [hot: %u, cold %u]", hot_store_version, cold_store_version);
  }

  phase = CheckpointPhase::RECOVER;
  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::REST)) {
    log_error("Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]",
              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
    return Status::Aborted;
  }
  return Status::Ok;
}

template<class K, class V, class D, class HHI, class CHI>
inline bool F2Kv<K, V, D, HHI, CHI>::CompactHotLog(uint64_t until_address, bool shift_begin_address, int n_threads) {
  return CompactLog(hot_store, StoreType::HOT, until_address, shift_begin_address, n_threads, false);
}

template<class K, class V, class D, class HHI, class CHI>
inline bool F2Kv<K, V, D, HHI, CHI>::CompactColdLog(uint64_t until_address, bool shift_begin_address, int n_threads) {
  return CompactLog(cold_store, StoreType::COLD, until_address, shift_begin_address, n_threads, false);
}

template<class K, class V, class D, class HHI, class CHI>
template <class S>

View on GitHub (pinned to 321d872eab)