What causes "Compact HOT log failed! :(" in microsoft/FASTER?

Retry the Checkpoint/compaction call — compaction is safe to re-run and transient contention is the usual cause.

Compact HOT log failed! :(

Error message

Compact HOT log failed! :(

What it means

The hot-log compaction step inside Checkpoint (CompactLog on the HOT store) returned false, meaning compaction did not complete successfully. The log is left uncompacted; the checkpoint itself may still proceed, but truncated/hot-log space was not reclaimed and a requested cold-store checkpoint may be affected.

Solutions

Retry the Checkpoint/compaction call — compaction is safe to re-run and transient contention is the usual cause.
Increase hot_log_compaction_config.num_threads (e.g. to match core count) so segment folds finish.
Check that until_address is within the flushed/safe region of the hot log before calling.
Reduce concurrent write load during compaction, or trigger compaction during a low-traffic window.

Example fix

// before
auto cfg = CompactionConfig{1}; // 1 thread
kv.CompactLog(until_address, 1);
// after
auto cfg = CompactionConfig{std::thread::hardware_concurrency()};
kv.CompactLog(until_address, cfg.num_threads); // retry on failure

Defensive patterns

Strategy: retry

Validate before calling

// before compacting
uint64_t until = kv.hlog.flushed_until_address; // must be within safe region
if (until == 0) return; // nothing flushed yet
if (cfg.num_threads < 2) cfg.num_threads = std::thread::hardware_concurrency();

Prevention

Size num_threads to core count for large logs.
Compact during low-traffic windows.
Validate until_address against the flushed safe address.
Re-run compaction on failure — it is idempotent and safe.

When it happens

Trigger: ShouldCompactHlog decides the hot log needs compaction (log size/usage exceeds hot_log_compaction_config thresholds) and CompactLog fails — typically because compaction threads could not finish within the iteration, the until_address was invalid, or the fold/scan over segments threw internally.

Common situations: Very small hot_log_compaction_config.num_threads on a huge log; compaction while heavy read/write traffic keeps shifting the safe address; until_address pointing beyond flushed data.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/c1e239a0445453b9. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/core/f2.h:934

      bool success = cold_store.InternalCheckpoint(nullptr, callback, checkpoint_.token,
                                                  static_cast<void*>(&checkpoint_));
      if (!success) {
        log_error("Cold store checkpoint failed!");
      }

      auto desired = success ? StoreCheckpointStatus::ACTIVE : StoreCheckpointStatus::FAILED;
      checkpoint_.cold_store_status.store(desired);
      log_debug("Cold store checkpoint issued! (status: %s)",
                STORE_CHECKPOINT_STATUS_STR[static_cast<int>(checkpoint_.cold_store_status)]);
    }

    // Hot-Cold compaction (optional cold-log checkpoint)
    if (ShouldCompactHlog(hot_store, StoreType::HOT, hot_log_compaction_config, until_address)) {
      hot_store.StartSession();
      cold_store.StartSession();
      if (!CompactLog(hot_store, StoreType::HOT, until_address, true,
                      hot_log_compaction_config.num_threads, should_checkpoint_cold_store())) {
        log_error("Compact HOT log failed! :(");
      }
      cold_store.StopSession();
      hot_store.StopSession();
    }

    // Cold-Cold compaction (optional cold-log checkpoint)
    if (ShouldCompactHlog(cold_store, StoreType::COLD, cold_log_compaction_config, until_address)) {
      cold_store.StartSession();
      if (!CompactLog(cold_store, StoreType::COLD, until_address, true,
                      cold_log_compaction_config.num_threads, should_checkpoint_cold_store())) {
        log_error("Compact COLD log failed!  :(");
      }
      cold_store.StopSession();
    }

    wait = (checkpoint_.hot_store_status != StoreCheckpointStatus::REQUESTED &&
            !ShouldCompactHlog(hot_store, StoreType::HOT, hot_log_compaction_config, until_address) &&
            !ShouldCompactHlog(cold_store, StoreType::COLD, cold_log_compaction_config, until_address));

View on GitHub (pinned to 321d872eab)