{"record":{"id":"03d8c2536b2c0d91","repo":"microsoft/FASTER","slug":"unexpected-checkpoint-status-for-cold-store-expected-s","errorCode":null,"errorMessage":"Unexpected checkpoint status for COLD store [expected: %s, actual: %s]","messagePattern":"Unexpected checkpoint status for COLD store \\[expected: (.+?), actual: (.+?)\\]","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cc/src/core/f2.h","lineNumber":903,"sourceCode":"        log_error(\"Hot store checkpoint failed!\");\n      }\n\n      auto desired = success ? StoreCheckpointStatus::ACTIVE : StoreCheckpointStatus::FAILED;\n      checkpoint_.hot_store_status.store(desired);\n      log_debug(\"Hot store checkpoint issued! (status: %s)\",\n                STORE_CHECKPOINT_STATUS_STR[static_cast<int>(checkpoint_.hot_store_status)]);\n    }\n\n    // Cold log checkpoint (high-priority)\n    if (should_checkpoint_cold_store() && checkpoint_.IsLazyCheckpointExpired()) {\n      auto callback = [](void* ctxt, Status result, uint64_t persistent_serial_num) {\n        auto checkpoint = static_cast<HotColdCheckpointState*>(ctxt);\n        assert(checkpoint->phase.load() == CheckpointPhase::COLD_STORE_CHECKPOINT);\n        assert(checkpoint->cold_store_status.load() == StoreCheckpointStatus::ACTIVE);\n\n        StoreCheckpointStatus status;\n        if ((status = checkpoint->cold_store_status.load()) != StoreCheckpointStatus::ACTIVE) {\n          log_error(\"Unexpected checkpoint status for COLD store [expected: %s, actual: %s]\",\n                    STORE_CHECKPOINT_STATUS_STR[static_cast<int>(StoreCheckpointStatus::ACTIVE)],\n                    STORE_CHECKPOINT_STATUS_STR[static_cast<int>(status)]);\n          assert(false);\n        }\n\n        // Set checkpoint status for cold-log\n        status = (result == Status::Ok) ? StoreCheckpointStatus::FINISHED : StoreCheckpointStatus::FAILED;\n        checkpoint->cold_store_status.store(status);\n        log_debug(\"Cold store checkpoint result: %s\", StatusStr(result));\n      };\n\n      log_debug(\"Issuing cold store checkpoint... (active sessions: %u)\", cold_store.NumActiveSessions());\n      bool success = cold_store.InternalCheckpoint(nullptr, callback, checkpoint_.token,\n                                                  static_cast<void*>(&checkpoint_));\n      if (!success) {\n        log_error(\"Cold store checkpoint failed!\");\n      }\n","sourceCodeStart":885,"sourceCodeEnd":921,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cc/src/core/f2.h#L885-L921","documentation":"During a hot-cold checkpoint, the callback that runs after the cold store's checkpoint completes asserts that the cold store's checkpoint status is ACTIVE. If the status read inside the COLD_STORE_CHECKPOINT phase is anything other than ACTIVE, this error is logged (and the internal assert fires in debug builds). It signals an internal race or inconsistency in checkpoint state tracking, not a user-input problem.","triggerScenarios":"The callback set via InternalCheckpoint on the cold store executes when checkpoint_.cold_store_status was not stored as StoreCheckpointStatus::ACTIVE — e.g. the cold store checkpoint was issued with a false/failing return that stored FAILED, or another thread mutated cold_store_status between the issue (error 181 path) and the callback run.","commonSituations":"Concurrent Checkpoint()/CompactLog() calls on the same FasterKv hot-cold pair without external serialization; a cold store whose InternalCheckpoint returned false (checkpoint init failed) while the phase state machine still reached COLD_STORE_CHECKPOINT; debug builds where the assert(false) crashes the process.","solutions":["Ensure only one thread invokes Checkpoint()/CompactLog() at a time (or protect it with a mutex); checkpoint state is not designed for concurrent issuers.","Check the return value of the cold store InternalCheckpoint / earlier log_error('Cold store checkpoint failed!') to find the root cause of the FAILED status.","Verify the checkpoint callback was registered before the status store and that cold_store_status is only written by the checkpoint coordinator.","Run a debug build to get the assert(false) location with full state, then file/investigate as an internal invariant violation if reproducible on a single-threaded checkpoint."],"exampleFix":"// before (concurrent callers)\nthread1: session.Checkpoint(...);\nthread2: session.CompactLog(...);\n// after\nstd::mutex cp_mu;\n{\n  std::lock_guard<std::mutex> g(cp_mu);\n  session.Checkpoint(...);\n}","handlingStrategy":"validation","validationCode":"// before issuing a checkpoint\nif (checkpoint_in_progress.exchange(true)) return; // serialize checkpoint issuers\ncold_store.StartSession(); // ensure active session on cold store","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never call Checkpoint()/CompactLog() concurrently from multiple threads.","Always pair StartSession/StopSession around checkpoint issuance.","Watch for the preceding 'Cold store checkpoint failed!' message — it is the root cause of the FAILED status.","Run debug builds in CI to surface the assert(false) invariant early."],"tags":["checkpoint","cold-store","race-condition","invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}