microsoft/FASTER · critical

Failed to recover hot store!

Error message

Failed to recover hot store!

What it means

During Recover(), the hot store failed to restore its state from the checkpoint directory/token and returned a non-Ok status, so the whole recovery is aborted. This means the persisted hot-store (FASTER) checkpoint data could not be loaded.

Solutions

1. Verify the checkpoint directory exists and contains a complete hot-store checkpoint for the given token
2. Confirm the token GUID matches a completed checkpoint (check earlier checkpoint logs for success)
3. Check disk/mount and file permissions for the checkpoint path
4. Fall back to the most recent completed checkpoint token, or start a fresh instance if no valid checkpoint exists

Example fix

// before
f2->Recover(bad_token, version, sessions); // token has no hot-store checkpoint
// after
if (!checkpoint_files_exist(dir, token)) { log_error("missing checkpoint"); return; }
Status s = f2->Recover(token, version, sessions);
if (s != Status::Ok) { /* fall back to previous token */ }

if (!std::filesystem::exists(ckpt_dir + "/hot/" + token.ToString())) { use_previous_token(); }

Status s = f2->Recover(token, v, ids);
if (s != Status::Ok) { token = last_completed_token(dir); s = f2->Recover(token, v, ids); }

When it happens

Trigger: Recover() called with a token pointing to missing, incomplete, or corrupt hot-store checkpoint files; disk I/O errors while reading the checkpoint; checkpoint directory not accessible/mounted.

Common situations: Recovering after an unclean shutdown where the checkpoint never finished; wrong or stale token GUID; storage volume not mounted or permissions changed; checkpoint files deleted by a cleanup job.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to recover cold store!
• Error reading page from device
• Hot store checkpoint failed!
• Unexpected entry type
• Unable to set first valid segment to

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/df33292518e5d78b. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:705

      assert(false);
    }
    log_debug("Requesting cold store checkpoint...");
    checkpoint->cold_store_status.store(StoreCheckpointStatus::REQUESTED);
  }
}

template<class K, class V, class D, class HHI, class CHI>
inline Status F2Kv<K, V, D, HHI, CHI>::Recover(const Guid& token, uint32_t& version, std::vector<Guid>& session_ids) {
  CheckpointPhase phase = CheckpointPhase::REST;
  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::RECOVER)) {
    log_error("Unexpected checkpoint phase during recovery [expected: REST, actual: %s]",
              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
    return Status::Aborted;
  }

  uint32_t hot_store_version, cold_store_version;
  if (hot_store.Recover(token, token, hot_store_version, session_ids) != Status::Ok) {
    log_error("Failed to recover hot store!");
    return Status::Aborted;
  }

  std::vector<Guid> temp_vector;
  if (cold_store.Recover(token, token, cold_store_version, temp_vector) != Status::Ok) {
    log_error("Failed to recover cold store!");
    return Status::Aborted;
  }

  if (hot_store_version != cold_store_version) {
    log_warn("Version of stores differ [hot: %u, cold %u]", hot_store_version, cold_store_version);
  }

  phase = CheckpointPhase::RECOVER;
  if (!checkpoint_.phase.compare_exchange_strong(phase, CheckpointPhase::REST)) {
    log_error("Unexpected checkpoint phase during recovery [expected: RECOVER, actual: %s]",
              CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
    return Status::Aborted;

View on GitHub (pinned to 321d872eab)