{"record":{"id":"c1e239a0445453b9","repo":"microsoft/FASTER","slug":"compact-hot-log-failed","errorCode":null,"errorMessage":"Compact HOT log failed! :(","messagePattern":"Compact HOT log failed! :\\(","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cc/src/core/f2.h","lineNumber":934,"sourceCode":"      bool success = cold_store.InternalCheckpoint(nullptr, callback, checkpoint_.token,\n                                                  static_cast<void*>(&checkpoint_));\n      if (!success) {\n        log_error(\"Cold store checkpoint failed!\");\n      }\n\n      auto desired = success ? StoreCheckpointStatus::ACTIVE : StoreCheckpointStatus::FAILED;\n      checkpoint_.cold_store_status.store(desired);\n      log_debug(\"Cold store checkpoint issued! (status: %s)\",\n                STORE_CHECKPOINT_STATUS_STR[static_cast<int>(checkpoint_.cold_store_status)]);\n    }\n\n    // Hot-Cold compaction (optional cold-log checkpoint)\n    if (ShouldCompactHlog(hot_store, StoreType::HOT, hot_log_compaction_config, until_address)) {\n      hot_store.StartSession();\n      cold_store.StartSession();\n      if (!CompactLog(hot_store, StoreType::HOT, until_address, true,\n                      hot_log_compaction_config.num_threads, should_checkpoint_cold_store())) {\n        log_error(\"Compact HOT log failed! :(\");\n      }\n      cold_store.StopSession();\n      hot_store.StopSession();\n    }\n\n    // Cold-Cold compaction (optional cold-log checkpoint)\n    if (ShouldCompactHlog(cold_store, StoreType::COLD, cold_log_compaction_config, until_address)) {\n      cold_store.StartSession();\n      if (!CompactLog(cold_store, StoreType::COLD, until_address, true,\n                      cold_log_compaction_config.num_threads, should_checkpoint_cold_store())) {\n        log_error(\"Compact COLD log failed!  :(\");\n      }\n      cold_store.StopSession();\n    }\n\n    wait = (checkpoint_.hot_store_status != StoreCheckpointStatus::REQUESTED &&\n            !ShouldCompactHlog(hot_store, StoreType::HOT, hot_log_compaction_config, until_address) &&\n            !ShouldCompactHlog(cold_store, StoreType::COLD, cold_log_compaction_config, until_address));","sourceCodeStart":916,"sourceCodeEnd":952,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cc/src/core/f2.h#L916-L952","documentation":"The hot-log compaction step inside Checkpoint (CompactLog on the HOT store) returned false, meaning compaction did not complete successfully. The log is left uncompacted; the checkpoint itself may still proceed, but truncated/hot-log space was not reclaimed and a requested cold-store checkpoint may be affected.","triggerScenarios":"ShouldCompactHlog decides the hot log needs compaction (log size/usage exceeds hot_log_compaction_config thresholds) and CompactLog fails — typically because compaction threads could not finish within the iteration, the until_address was invalid, or the fold/scan over segments threw internally.","commonSituations":"Very small hot_log_compaction_config.num_threads on a huge log; compaction while heavy read/write traffic keeps shifting the safe address; until_address pointing beyond flushed data.","solutions":["Retry the Checkpoint/compaction call — compaction is safe to re-run and transient contention is the usual cause.","Increase hot_log_compaction_config.num_threads (e.g. to match core count) so segment folds finish.","Check that until_address is within the flushed/safe region of the hot log before calling.","Reduce concurrent write load during compaction, or trigger compaction during a low-traffic window."],"exampleFix":"// before\nauto cfg = CompactionConfig{1}; // 1 thread\nkv.CompactLog(until_address, 1);\n// after\nauto cfg = CompactionConfig{std::thread::hardware_concurrency()};\nkv.CompactLog(until_address, cfg.num_threads); // retry on failure","handlingStrategy":"retry","validationCode":"// before compacting\nuint64_t until = kv.hlog.flushed_until_address; // must be within safe region\nif (until == 0) return; // nothing flushed yet\nif (cfg.num_threads < 2) cfg.num_threads = std::thread::hardware_concurrency();","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size num_threads to core count for large logs.","Compact during low-traffic windows.","Validate until_address against the flushed safe address.","Re-run compaction on failure — it is idempotent and safe."],"tags":["compaction","hot-store","hybridlog","checkpoint"],"backgroundTag":"database-query-failed","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}