{"record":{"id":"c8e1893de5734b88","repo":"microsoft/FASTER","slug":"unexpected-checkpoint-phase-during-recovery-expected-rest","errorCode":null,"errorMessage":"Unexpected checkpoint phase during recovery [expected: REST, actual: %s]","messagePattern":"Unexpected checkpoint phase during recovery \\[expected: REST, actual: (.+?)\\]","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cc/src/core/f2.h","lineNumber":698,"sourceCode":"    }\n    log_debug(\"Moving to cold-store checkpoint phase\");\n    checkpoint->phase.store(CheckpointPhase::COLD_STORE_CHECKPOINT);\n\n    if ((status = checkpoint->cold_store_status.load()) != StoreCheckpointStatus::IDLE) {\n      log_error(\"Unexpected checkpoint status for COLD store [expected: IDLE, actual: %s]\",\n                STORE_CHECKPOINT_STATUS_STR[static_cast<int>(status)]);\n      assert(false);\n    }\n    log_debug(\"Requesting cold store checkpoint...\");\n    checkpoint->cold_store_status.store(StoreCheckpointStatus::REQUESTED);\n  }\n}\n\ntemplate<class K, class V, class D, class HHI, class CHI>\ninline Status F2Kv<K, V, D, HHI, CHI>::Recover(const Guid& token, uint32_t& version, std::vector<Guid>& session_ids) {\n  CheckpointPhase phase = CheckpointPhase::REST;\n  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::RECOVER)) {\n    log_error(\"Unexpected checkpoint phase during recovery [expected: REST, actual: %s]\",\n              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);\n    return Status::Aborted;\n  }\n\n  uint32_t hot_store_version, cold_store_version;\n  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {\n    log_error(\"Failed to recover hot store!\");\n    return Status::Aborted;\n  }\n\n  std::vector<Guid> temp_vector;\n  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {\n    log_error(\"Failed to recover cold store!\");\n    return Status::Aborted;\n  }\n\n  if (hot_store_version != cold_store_version) {\n    log_warn(\"Version of stores differ [hot: %u, cold %u]\", hot_store_version, cold_store_version);","sourceCodeStart":680,"sourceCodeEnd":716,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cc/src/core/f2.h#L680-L716","documentation":"Recover() must start from the REST phase; it atomically swaps phase REST -> RECOVER and returns Status::Aborted if the current phase is anything else. This prevents recovery from racing with an in-progress checkpoint, which would read inconsistent store state.","triggerScenarios":"Calling Recover(token, ...) while a checkpoint is in progress (phase in HOT/COLD_STORE_CHECKPOINT) or while another Recover is already running.","commonSituations":"Calling Recover concurrently from multiple sessions; issuing recovery right after a Checkpoint() that hasn't finished; application startup code racing a background checkpoint thread.","solutions":["Ensure no checkpoint is in flight before calling Recover","Only call Recover once during session setup and from a single thread","Retry Recover after the current checkpoint completes and phase returns to REST","Check the returned Status::Aborted and re-attempt later instead of ignoring it"],"exampleFix":"// before\nf2->Checkpoint();\nf2->Recover(token, version, sessions); // may abort: checkpoint in flight\n// after\nwait_for_checkpoint_completion(f2);\nStatus s = f2->Recover(token, version, sessions);\nif (s != Status::Ok) { /* retry later */ }","handlingStrategy":"retry","validationCode":"// ensure no checkpoint active: app-level flag set around Checkpoint()\nif (checkpointing.load()) { wait_until_rest(); }","typeGuard":null,"tryCatchPattern":"Status s = f2->Recover(token, v, ids);\nif (s == Status::Aborted) { /* checkpoint in flight; retry after it completes */ }","preventionTips":["Call Recover only during startup, once, from one thread","Do not mix Checkpoint() and Recover() in time"],"tags":["recovery","checkpoint","state-machine"],"backgroundTag":"invalid-state-transition","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}