Unexpected checkpoint phase

Error message

Unexpected checkpoint phase [expected: HOT_STORE_CHECKPOINT, actual: %s]

What it means

When a checkpoint finishes the hot-store portion, the driver transitions phase from HOT_STORE_CHECKPOINT to COLD_STORE_CHECKPOINT. If the phase is not HOT_STORE_CHECKPOINT at that point, the checkpoint state machine was entered from an unexpected phase, indicating concurrent checkpoints or a corrupted transition sequence.

Solutions

Serialize Checkpoint() calls so only one runs at a time
Verify phase == REST before issuing a new checkpoint
Inspect prior logs for earlier 'Unexpected checkpoint phase' errors to locate the first bad transition
Reset the instance (re-open F2) to clear the stuck state

Example fix

// before
f2->Checkpoint(); // issued while previous checkpoint still in flight
// after
if (prev_checkpoint_done.load()) { f2->Checkpoint(); prev_checkpoint_done.store(false); }

Defensive patterns

Strategy: validation

Validate before calling

if (checkpoint_in_flight) { throw std::runtime_error("checkpoint already in progress"); }

Prevention

Single-writer checkpoint initiation
Complete each checkpoint round fully before the next
Monitor logs for early phase-mismatch errors

When it happens

Trigger: Two threads calling Checkpoint() concurrently so the second overwrites the phase; issuing a checkpoint while the machine is already in COLD_STORE_CHECKPOINT or RECOVER phase.

Common situations: Overlapping checkpoint requests from different client threads; a hot-store checkpoint callback arriving after a manual phase reset.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/1a3bec68a0de7ff5. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:678

    }
  }

  if (--checkpoint->threads_pending_hot_store_persist == 0) {
    // All threads finished checkpointing hot store
    // Last thread responsible for moving to next phase (i.e., cold-store checkpointing)
    StoreCheckpointStatus status;

    // Mark hot store checkpoint status to finished (or failed)
    Status global_result = checkpoint->hot_store_checkpoint_result.load();
    log_debug("Hot store checkpoint result: %s", StatusStr(global_result));

    status = (global_result == Status::Ok) ? StoreCheckpointStatus::FINISHED : StoreCheckpointStatus::FAILED;
    checkpoint->hot_store_status.store(status);

    // Request cold store checkpointing
    CheckpointPhase phase;
    if ((phase = checkpoint->phase.load()) != CheckpointPhase::HOT_STORE_CHECKPOINT) {
      log_error("Unexpected checkpoint phase [expected: HOT_STORE_CHECKPOINT, actual: %s]",
                CHECKPOINT_PHASE_STR[static_cast<int>(phase)]);
    }
    log_debug("Moving to cold-store checkpoint phase");
    checkpoint->phase.store(CheckpointPhase::COLD_STORE_CHECKPOINT);

    if ((status = checkpoint->cold_store_status.load()) != StoreCheckpointStatus::IDLE) {
      log_error("Unexpected checkpoint status for COLD store [expected: IDLE, actual: %s]",
                STORE_CHECKPOINT_STATUS_STR[static_cast<int>(status)]);
      assert(false);
    }
    log_debug("Requesting cold store checkpoint...");
    checkpoint->cold_store_status.store(StoreCheckpointStatus::REQUESTED);
  }
}

template<class K, class V, class D, class HHI, class CHI>
inline Status F2Kv<K, V, D, HHI, CHI>::Recover(const Guid& token, uint32_t& version, std::vector<Guid>& session_ids) {
  CheckpointPhase phase = CheckpointPhase::REST;

View on GitHub (pinned to 321d872eab)