microsoft/FASTER · critical

Failed to recover cold store!

Error message

Failed to recover cold store!

What it means

During Recover(), the cold store failed to restore its state from the checkpoint data, aborting recovery. The hot store may already be recovered, but the tiered instance requires both stores to load successfully.

Solutions

1. Verify the cold-store checkpoint files exist and are complete for the token
2. Check the cold-store storage path is mounted and writable/readable
3. Watch for the 'Version of stores differ' warning right after — it signals a partially completed checkpoint; prefer a token where versions match
4. Fall back to the latest fully completed checkpoint or re-initialize if none exists

Example fix

// before
f2->Recover(token, version, sessions); // cold store files missing
// after
if (!cold_store_checkpoint_complete(dir, token)) { token = last_complete_checkpoint(dir); }
Status s = f2->Recover(token, version, sessions);

if (!cold_store_checkpoint_complete(dir, token)) { token = last_complete_token(dir); }

Status s = f2->Recover(token, v, ids);
if (s != Status::Ok) { /* fall back to an earlier complete checkpoint */ }

When it happens

Trigger: Recover() called with a token whose cold-store (hybridLog to cold tier) checkpoint files are missing, truncated, or unreadable; cold-store storage unavailable; version mismatch indicating incomplete checkpoint pair.

Common situations: Cold-tier storage volume not mounted; checkpoint captured mid-write (store died during Checkpoint); cold-store files cleaned up or moved between runs.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to recover hot store!
• Error reading page from device
• Unexpected entry type
• Unable to set first valid segment to
• Unable to set last valid segment to

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/291264e4948df085. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:711

template<class K, class V, class D, class HHI, class CHI>
inline Status F2Kv<K, V, D, HHI, CHI>::Recover(const Guid& token, uint32_t& version, std::vector<Guid>& session_ids) {
  CheckpointPhase phase = CheckpointPhase::REST;
  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::RECOVER)) {
    log_error("Unexpected checkpoint phase during recovery [expected: REST, actual: %s]",
              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
    return Status::Aborted;
  }

  uint32_t hot_store_version, cold_store_version;
  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {
    log_error("Failed to recover hot store!");
    return Status::Aborted;
  }

  std::vector<Guid> temp_vector;
  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {
    log_error("Failed to recover cold store!");
    return Status::Aborted;
  }

  if (hot_store_version != cold_store_version) {
    log_warn("Version of stores differ [hot: %u, cold %u]", hot_store_version, cold_store_version);
  }

  phase = CheckpointPhase::RECOVER;
  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::REST)) {
    log_error("Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]",
              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
    return Status::Aborted;
  }
  return Status::Ok;
}

template<class K, class V, class D, class HHI, class CHI>
inline bool F2Kv<K, V, D, HHI, CHI>::CompactHotLog(uint64_t until_address, bool shift_begin_address, int n_threads) {

View on GitHub (pinned to 321d872eab)