{"record":{"id":"291264e4948df085","repo":"microsoft/FASTER","slug":"failed-to-recover-cold-store","errorCode":null,"errorMessage":"Failed to recover cold store!","messagePattern":"Failed to recover cold store!","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cc/src/core/f2.h","lineNumber":711,"sourceCode":"\ntemplate<class K, class V, class D, class HHI, class CHI>\ninline Status F2Kv<K, V, D, HHI, CHI>::Recover(const Guid& token, uint32_t& version, std::vector<Guid>& session_ids) {\n  CheckpointPhase phase = CheckpointPhase::REST;\n  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::RECOVER)) {\n    log_error(\"Unexpected checkpoint phase during recovery [expected: REST, actual: %s]\",\n              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);\n    return Status::Aborted;\n  }\n\n  uint32_t hot_store_version, cold_store_version;\n  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {\n    log_error(\"Failed to recover hot store!\");\n    return Status::Aborted;\n  }\n\n  std::vector<Guid> temp_vector;\n  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {\n    log_error(\"Failed to recover cold store!\");\n    return Status::Aborted;\n  }\n\n  if (hot_store_version != cold_store_version) {\n    log_warn(\"Version of stores differ [hot: %u, cold %u]\", hot_store_version, cold_store_version);\n  }\n\n  phase = CheckpointPhase::RECOVER;\n  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::REST)) {\n    log_error(\"Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]\",\n              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);\n    return Status::Aborted;\n  }\n  return Status::Ok;\n}\n\ntemplate<class K, class V, class D, class HHI, class CHI>\ninline bool F2Kv<K, V, D, HHI, CHI>::CompactHotLog(uint64_t until_address, bool shift_begin_address, int n_threads) {","sourceCodeStart":693,"sourceCodeEnd":729,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cc/src/core/f2.h#L693-L729","documentation":"During Recover(), the cold store failed to restore its state from the checkpoint data, aborting recovery. The hot store may already be recovered, but the tiered instance requires both stores to load successfully.","triggerScenarios":"Recover() called with a token whose cold-store (hybridLog to cold tier) checkpoint files are missing, truncated, or unreadable; cold-store storage unavailable; version mismatch indicating incomplete checkpoint pair.","commonSituations":"Cold-tier storage volume not mounted; checkpoint captured mid-write (store died during Checkpoint); cold-store files cleaned up or moved between runs.","solutions":["Verify the cold-store checkpoint files exist and are complete for the token","Check the cold-store storage path is mounted and writable/readable","Watch for the 'Version of stores differ' warning right after — it signals a partially completed checkpoint; prefer a token where versions match","Fall back to the latest fully completed checkpoint or re-initialize if none exists"],"exampleFix":"// before\nf2->Recover(token, version, sessions); // cold store files missing\n// after\nif (!cold_store_checkpoint_complete(dir, token)) { token = last_complete_checkpoint(dir); }\nStatus s = f2->Recover(token, version, sessions);","handlingStrategy":"fallback","validationCode":"if (!cold_store_checkpoint_complete(dir, token)) { token = last_complete_token(dir); }","typeGuard":null,"tryCatchPattern":"Status s = f2->Recover(token, v, ids);\nif (s != Status::Ok) { /* fall back to an earlier complete checkpoint */ }","preventionTips":["Match hot/cold checkpoint versions before recovering","Ensure cold-tier storage is available at startup","Never delete checkpoint directories while the instance may recover"],"tags":["recovery","cold-store","persistence","io"],"backgroundTag":"file-read-failed","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}