microsoft/FASTER · error

Unexpected checkpoint phase

Error message

Unexpected checkpoint phase [expected: COLD_STORE_CHECKPOINT, actual: %s]

What it means

F2Kv (hot/cold tiered key-value store) runs a multi-phase checkpoint state machine in checkpoint_.phase. After all threads have issued their checkpoint callback, the phase must be COLD_STORE_CHECKPOINT before resetting to REST; if it is anything else, concurrent or skipped phase transitions corrupted the state machine, so this diagnostic is logged (and asserts in debug builds).

Solutions

1. Ensure only one Checkpoint() call is in flight at a time (serialize with an application-level mutex or a dedicated checkpoint thread)
2. Wait for a previous checkpoint to fully complete (phase returns to REST) before starting another
3. Restart/reload the F2 instance to reset checkpoint state after the corruption
4. Rebuild with asserts enabled (NDEBUG undefined) and reproduce under a race detector to find the racy caller

Example fix

// before
if (need_checkpoint) f2->Checkpoint(); // called from any worker thread
// after
static std::mutex ckpt_mu;
if (need_checkpoint) { std::lock_guard<std::mutex> g(ckpt_mu); f2->Checkpoint(); }

if (checkpoint_in_flight.exchange(true)) { skip_or_wait(); }

When it happens

Trigger: Calling Checkpoint() from multiple sessions/threads concurrently, or aborting/handling a checkpoint mid-flight so the phase advances past COLD_STORE_CHECKPOINT before the last thread issues its callback.

Common situations: Multi-threaded clients issuing Checkpoint() at the same time; mixing Checkpoint() with recovery; a prior checkpoint left the state machine in a non-REST phase.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Unexpected checkpoint phase
• Unexpected checkpoint status for COLD store
• Invalid Enum Argument
• Unexpected checkpoint phase during recovery
• Unexpected checkpoint phase during recovery

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/a258df3e4a876a74. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:606

      return;
    }

    // call user-provided callback for this active session (if not already)
    uint64_t psn = checkpoint_.persistent_serial_nums[Thread::id()].load();
    if (psn == 0) {
      return;
    }
    Status result = (checkpoint_.hot_store_status.load() == StoreCheckpointStatus::FINISHED &&
                      checkpoint_.cold_store_status.load() == StoreCheckpointStatus::FINISHED)
                        ? Status::Ok : Status::IOError;
    checkpoint_.store_persistence_callback(result, psn);

    checkpoint_.persistent_serial_nums[Thread::id()].store(0);
    if (--checkpoint_.threads_pending_issue_callback == 0) {
      // all threads issued their callback -- atomically move to REST phase
      CheckpointPhase phase;
      if ((phase = checkpoint_.phase.load()) != CheckpointPhase::COLD_STORE_CHECKPOINT) {
        log_error("Unexpected checkpoint phase [expected: COLD_STORE_CHECKPOINT, actual: %s]",
                  CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
        assert(false);
      }

      log_debug("Hot-cold checkpoint finished! Moving to REST phase");
      checkpoint_.Reset();
      checkpoint_.phase.store(CheckpointPhase::REST);
    }
  }
}

template<class K, class V, class D, class HHI, class CHI>
inline bool F2Kv<K, V, D, HHI, CHI>::Checkpoint(HybridLogPersistenceCallback hybrid_log_persistence_callback,
                                                Guid& token, bool lazy) {
  CheckpointPhase expected_phase = CheckpointPhase::REST;
  if (!checkpoint_.phase.compare_exchange_strong(expected_phase, CheckpointPhase::HOT_STORE_CHECKPOINT)) {
    throw std::runtime_error{ "Can start checkpoint only when no other concurrent op" };
  }

View on GitHub (pinned to 321d872eab)