{"record":{"id":"60dd732a7fb01b6b","repo":"microsoft/FASTER","slug":"unexpected-checkpoint-phase-during-recovery-expected-recover","errorCode":null,"errorMessage":"Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]","messagePattern":"Unexpected checkpoint phase during recovery \\[expected: RECOVER, actual: (.+?)\\]","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cc/src/core/f2.h","lineNumber":721,"sourceCode":"  uint32_t hot_store_version, cold_store_version;\n  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {\n    log_error(\"Failed to recover hot store!\");\n    return Status::Aborted;\n  }\n\n  std::vector<Guid> temp_vector;\n  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {\n    log_error(\"Failed to recover cold store!\");\n    return Status::Aborted;\n  }\n\n  if (hot_store_version != cold_store_version) {\n    log_warn(\"Version of stores differ [hot: %u, cold %u]\", hot_store_version, cold_store_version);\n  }\n\n  phase = CheckpointPhase::RECOVER;\n  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::REST)) {\n    log_error(\"Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]\",\n              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);\n    return Status::Aborted;\n  }\n  return Status::Ok;\n}\n\ntemplate<class K, class V, class D, class HHI, class CHI>\ninline bool F2Kv<K, V, D, HHI, CHI>::CompactHotLog(uint64_t until_address, bool shift_begin_address, int n_threads) {\n  return CompactLog(hot_store, StoreType::HOT, until_address, shift_begin_address, n_threads, false);\n}\n\ntemplate<class K, class V, class D, class HHI, class CHI>\ninline bool F2Kv<K, V, D, HHI, CHI>::CompactColdLog(uint64_t until_address, bool shift_begin_address, int n_threads) {\n  return CompactLog(cold_store, StoreType::COLD, until_address, shift_begin_address, n_threads, false);\n}\n\ntemplate<class K, class V, class D, class HHI, class CHI>\ntemplate <class S>","sourceCodeStart":703,"sourceCodeEnd":739,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cc/src/core/f2.h#L703-L739","documentation":"At the end of Recover(), the phase must be RECOVER before being swapped back to REST. If the CAS fails, another thread altered the checkpoint phase during recovery (e.g., started a checkpoint concurrently), so recovery returns Status::Aborted even though stores may have loaded.","triggerScenarios":"Calling Checkpoint() (or a second Recover) from another thread while Recover() is executing; background checkpoint logic interleaving with recovery.","commonSituations":"Startup code that triggers recovery on one thread while a maintenance thread kicks off a checkpoint; duplicated session-setup code paths calling Recover twice.","solutions":["Ensure no other thread calls Checkpoint()/Recover() during recovery","Call Recover exactly once per session, from a single thread, before starting any checkpoint activity","Treat the returned Status::Aborted as a real failure and redo recovery rather than proceeding","Re-open the instance to return the state machine to REST before retrying"],"exampleFix":"// before\nstd::thread(recover_fn); std::thread(checkpoint_fn); // both mutate phase\n// after\nrecover_fn(); // complete recovery first\nstd::thread(checkpoint_fn); // then checkpoint","handlingStrategy":"try-catch","validationCode":"std::lock_guard<std::mutex> g(recover_mu); // prevents concurrent phase mutation","typeGuard":null,"tryCatchPattern":"Status s = f2->Recover(token, v, ids);\nif (s == Status::Aborted) { /* reopen or re-run recovery single-threaded */ }","preventionTips":["Recover from a single thread before any background workers start","Never call Checkpoint() concurrently with Recover()"],"tags":["recovery","state-machine","concurrency"],"backgroundTag":"invalid-state-transition","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}