{"record":{"id":"df33292518e5d78b","repo":"microsoft/FASTER","slug":"failed-to-recover-hot-store","errorCode":null,"errorMessage":"Failed to recover hot store!","messagePattern":"Failed to recover hot store!","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cc/src/core/f2.h","lineNumber":705,"sourceCode":"      assert(false);\n    }\n    log_debug(\"Requesting cold store checkpoint...\");\n    checkpoint->cold_store_status.store(StoreCheckpointStatus::REQUESTED);\n  }\n}\n\ntemplate<class K, class V, class D, class HHI, class CHI>\ninline Status F2Kv<K, V, D, HHI, CHI>::Recover(const Guid& token, uint32_t& version, std::vector<Guid>& session_ids) {\n  CheckpointPhase phase = CheckpointPhase::REST;\n  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::RECOVER)) {\n    log_error(\"Unexpected checkpoint phase during recovery [expected: REST, actual: %s]\",\n              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);\n    return Status::Aborted;\n  }\n\n  uint32_t hot_store_version, cold_store_version;\n  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {\n    log_error(\"Failed to recover hot store!\");\n    return Status::Aborted;\n  }\n\n  std::vector<Guid> temp_vector;\n  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {\n    log_error(\"Failed to recover cold store!\");\n    return Status::Aborted;\n  }\n\n  if (hot_store_version != cold_store_version) {\n    log_warn(\"Version of stores differ [hot: %u, cold %u]\", hot_store_version, cold_store_version);\n  }\n\n  phase = CheckpointPhase::RECOVER;\n  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::REST)) {\n    log_error(\"Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]\",\n              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);\n    return Status::Aborted;","sourceCodeStart":687,"sourceCodeEnd":723,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cc/src/core/f2.h#L687-L723","documentation":"During Recover(), the hot store failed to restore its state from the checkpoint directory/token and returned a non-Ok status, so the whole recovery is aborted. This means the persisted hot-store (FASTER) checkpoint data could not be loaded.","triggerScenarios":"Recover() called with a token pointing to missing, incomplete, or corrupt hot-store checkpoint files; disk I/O errors while reading the checkpoint; checkpoint directory not accessible/mounted.","commonSituations":"Recovering after an unclean shutdown where the checkpoint never finished; wrong or stale token GUID; storage volume not mounted or permissions changed; checkpoint files deleted by a cleanup job.","solutions":["Verify the checkpoint directory exists and contains a complete hot-store checkpoint for the given token","Confirm the token GUID matches a completed checkpoint (check earlier checkpoint logs for success)","Check disk/mount and file permissions for the checkpoint path","Fall back to the most recent completed checkpoint token, or start a fresh instance if no valid checkpoint exists"],"exampleFix":"// before\nf2->Recover(bad_token, version, sessions); // token has no hot-store checkpoint\n// after\nif (!checkpoint_files_exist(dir, token)) { log_error(\"missing checkpoint\"); return; }\nStatus s = f2->Recover(token, version, sessions);\nif (s != Status::Ok) { /* fall back to previous token */ }","handlingStrategy":"fallback","validationCode":"if (!std::filesystem::exists(ckpt_dir + \"/hot/\" + token.ToString())) { use_previous_token(); }","typeGuard":null,"tryCatchPattern":"Status s = f2->Recover(token, v, ids);\nif (s != Status::Ok) { token = last_completed_token(dir); s = f2->Recover(token, v, ids); }","preventionTips":["Only recover from tokens of fully completed checkpoints","Verify checkpoint files exist and are readable before Recover","Keep the checkpoint volume mounted and backed up"],"tags":["recovery","hot-store","persistence","io"],"backgroundTag":"file-read-failed","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}