{"record":{"id":"a258df3e4a876a74","repo":"microsoft/FASTER","slug":"unexpected-checkpoint-phase-expected-cold-store-checkpoint","errorCode":null,"errorMessage":"Unexpected checkpoint phase [expected: COLD_STORE_CHECKPOINT, actual: %s]","messagePattern":"Unexpected checkpoint phase \\[expected: COLD_STORE_CHECKPOINT, actual: (.+?)\\]","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cc/src/core/f2.h","lineNumber":606,"sourceCode":"      return;\n    }\n\n    // call user-provided callback for this active session (if not already)\n    uint64_t psn = checkpoint_.persistent_serial_nums[Thread::id()].load();\n    if (psn == 0) {\n      return;\n    }\n    Status result = (checkpoint_.hot_store_status.load() == StoreCheckpointStatus::FINISHED &&\n                      checkpoint_.cold_store_status.load() == StoreCheckpointStatus::FINISHED)\n                        ? Status::Ok : Status::IOError;\n    checkpoint_.store_persistence_callback(result, psn);\n\n    checkpoint_.persistent_serial_nums[Thread::id()].store(0);\n    if (--checkpoint_.threads_pending_issue_callback == 0) {\n      // all threads issued their callback -- atomically move to REST phase\n      CheckpointPhase phase;\n      if ((phase = checkpoint_.phase.load()) != CheckpointPhase::COLD_STORE_CHECKPOINT) {\n        log_error(\"Unexpected checkpoint phase [expected: COLD_STORE_CHECKPOINT, actual: %s]\",\n                  CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);\n        assert(false);\n      }\n\n      log_debug(\"Hot-cold checkpoint finished! Moving to REST phase\");\n      checkpoint_.Reset();\n      checkpoint_.phase.store(CheckpointPhase::REST);\n    }\n  }\n}\n\ntemplate<class K, class V, class D, class HHI, class CHI>\ninline bool F2Kv<K, V, D, HHI, CHI>::Checkpoint(HybridLogPersistenceCallback hybrid_log_persistence_callback,\n                                                Guid& token, bool lazy) {\n  CheckpointPhase expected_phase = CheckpointPhase::REST;\n  if (!checkpoint_.phase.compare_exchange_strong(expected_phase, CheckpointPhase::HOT_STORE_CHECKPOINT)) {\n    throw std::runtime_error{ \"Can start checkpoint only when no other concurrent op\" };\n  }","sourceCodeStart":588,"sourceCodeEnd":624,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cc/src/core/f2.h#L588-L624","documentation":"F2Kv (hot/cold tiered key-value store) runs a multi-phase checkpoint state machine in checkpoint_.phase. After all threads have issued their checkpoint callback, the phase must be COLD_STORE_CHECKPOINT before resetting to REST; if it is anything else, concurrent or skipped phase transitions corrupted the state machine, so this diagnostic is logged (and asserts in debug builds).","triggerScenarios":"Calling Checkpoint() from multiple sessions/threads concurrently, or aborting/handling a checkpoint mid-flight so the phase advances past COLD_STORE_CHECKPOINT before the last thread issues its callback.","commonSituations":"Multi-threaded clients issuing Checkpoint() at the same time; mixing Checkpoint() with recovery; a prior checkpoint left the state machine in a non-REST phase.","solutions":["Ensure only one Checkpoint() call is in flight at a time (serialize with an application-level mutex or a dedicated checkpoint thread)","Wait for a previous checkpoint to fully complete (phase returns to REST) before starting another","Restart/reload the F2 instance to reset checkpoint state after the corruption","Rebuild with asserts enabled (NDEBUG undefined) and reproduce under a race detector to find the racy caller"],"exampleFix":"// before\nif (need_checkpoint) f2->Checkpoint(); // called from any worker thread\n// after\nstatic std::mutex ckpt_mu;\nif (need_checkpoint) { std::lock_guard<std::mutex> g(ckpt_mu); f2->Checkpoint(); }","handlingStrategy":"validation","validationCode":"if (checkpoint_in_flight.exchange(true)) { skip_or_wait(); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never call Checkpoint() concurrently from multiple threads","Track checkpoint completion with an atomic flag before re-issuing","Wait for phase REST between checkpoint rounds"],"tags":["checkpoint","state-machine","concurrency","internal-state"],"backgroundTag":"invalid-state-transition","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}