microsoft/FASTER · error

Unexpected checkpoint status for COLD store

Error message

Unexpected checkpoint status for COLD store [expected: %s, actual: %s]

What it means

During a hot-cold checkpoint, the callback that runs after the cold store's checkpoint completes asserts that the cold store's checkpoint status is ACTIVE. If the status read inside the COLD_STORE_CHECKPOINT phase is anything other than ACTIVE, this error is logged (and the internal assert fires in debug builds). It signals an internal race or inconsistency in checkpoint state tracking, not a user-input problem.

Solutions

1. Ensure only one thread invokes Checkpoint()/CompactLog() at a time (or protect it with a mutex); checkpoint state is not designed for concurrent issuers.
2. Check the return value of the cold store InternalCheckpoint / earlier log_error('Cold store checkpoint failed!') to find the root cause of the FAILED status.
3. Verify the checkpoint callback was registered before the status store and that cold_store_status is only written by the checkpoint coordinator.
4. Run a debug build to get the assert(false) location with full state, then file/investigate as an internal invariant violation if reproducible on a single-threaded checkpoint.

Example fix

// before (concurrent callers)
thread1: session.Checkpoint(...);
thread2: session.CompactLog(...);
// after
std::mutex cp_mu;
{
  std::lock_guard<std::mutex> g(cp_mu);
  session.Checkpoint(...);
}

// before issuing a checkpoint
if (checkpoint_in_progress.exchange(true)) return; // serialize checkpoint issuers
cold_store.StartSession(); // ensure active session on cold store

When it happens

Trigger: The callback set via InternalCheckpoint on the cold store executes when checkpoint_.cold_store_status was not stored as StoreCheckpointStatus::ACTIVE — e.g. the cold store checkpoint was issued with a false/failing return that stored FAILED, or another thread mutated cold_store_status between the issue (error 181 path) and the callback run.

Common situations: Concurrent Checkpoint()/CompactLog() calls on the same FasterKv hot-cold pair without external serialization; a cold store whose InternalCheckpoint returned false (checkpoint init failed) while the phase state machine still reached COLD_STORE_CHECKPOINT; debug builds where the assert(false) crashes the process.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Unexpected checkpoint status for COLD store
• Unexpected checkpoint status
• Cold store checkpoint failed!
• Compact COLD log failed! :(
• Unexpected sealed buffer found

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/03d8c2536b2c0d91. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:903

        log_error("Hot store checkpoint failed!");
      }

      auto desired = success ? StoreCheckpointStatus::ACTIVE : StoreCheckpointStatus::FAILED;
      checkpoint_.hot_store_status.store(desired);
      log_debug("Hot store checkpoint issued! (status: %s)",
                STORE_CHECKPOINT_STATUS_STR[static_cast<int>(checkpoint_.hot_store_status)]);
    }

    // Cold log checkpoint (high-priority)
    if (should_checkpoint_cold_store() && checkpoint_.IsLazyCheckpointExpired()) {
      auto callback = [](void* ctxt, Status result, uint64_t persistent_serial_num) {
        auto checkpoint = static_cast<HotColdCheckpointState*>(ctxt);
        assert(checkpoint->phase.load() == CheckpointPhase::COLD_STORE_CHECKPOINT);
        assert(checkpoint->cold_store_status.load() == StoreCheckpointStatus::ACTIVE);

        StoreCheckpointStatus status;
        if ((status = checkpoint->cold_store_status.load()) != StoreCheckpointStatus::ACTIVE) {
          log_error("Unexpected checkpoint status for COLD store [expected: %s, actual: %s]",
                    STORE_CHECKPOINT_STATUS_STR[static_cast<int>(StoreCheckpointStatus::ACTIVE)],
                    STORE_CHECKPOINT_STATUS_STR[static_cast<int>(status)]);
          assert(false);
        }

        // Set checkpoint status for cold-log
        status = (result == Status::Ok) ? StoreCheckpointStatus::FINISHED : StoreCheckpointStatus::FAILED;
        checkpoint->cold_store_status.store(status);
        log_debug("Cold store checkpoint result: %s", StatusStr(result));
      };

      log_debug("Issuing cold store checkpoint... (active sessions: %u)", cold_store.NumActiveSessions());
      bool success = cold_store.InternalCheckpoint(nullptr, callback, checkpoint_.token,
                                                  static_cast<void*>(&checkpoint_));
      if (!success) {
        log_error("Cold store checkpoint failed!");
      }

View on GitHub (pinned to 321d872eab)