What causes "Log data hashes differ! Something is tragically wrong!" in phacility/phabricator?

Re-run the archival when no builds are running (pause the daemon / let builds finish) so logs are quiescent.

phacility/phabricator · critical · Exception

Log data hashes differ! Something is tragically wrong!

Error message

Log data hashes differ! Something is tragically wrong!

What it means

Data-integrity check inside archive-logs: after re-writing a log's chunks in the new mode, the workflow recomputes a hash of the reconstructed log data and compares it with the hash before archival. A mismatch means the byte content changed during the rewrite — the archival did not round-trip — and the command aborts loudly rather than let silent corruption spread. Tragic, and meant to be.

Source

Thrown at src/applications/harbormaster/management/HarbormasterManagementArchiveLogsWorkflow.php:145

    echo tsprintf(
      "    %s\n",
      pht(
        '%s: %s -> %s',
        pht('Stored Chunks'),
        new PhutilNumber($old_stats['chunks']),
        new PhutilNumber($new_stats['chunks'])));

    echo tsprintf(
      "    %s\n",
      pht(
        '%s: %s -> %s',
        pht('Data Hash'),
        $old_stats['hash'],
        $new_stats['hash']));

    if ($old_stats['hash'] !== $new_stats['hash']) {
      throw new Exception(
        pht('Log data hashes differ! Something is tragically wrong!'));
    }
  }

}

View on GitHub (pinned to 5720a38cfe)

Solutions

Re-run the archival when no builds are running (pause the daemon / let builds finish) so logs are quiescent.
Run with --verbose/--details and inspect the reported old/new chunk counts and hashes to see whether the log grew (data appended) or truly differs.
Check the harbormaster_buildlogchunk table (or file storage) for the affected log ID for truncated or overlapping chunks; repair from the original storage if available.
Ensure only one archive-logs process runs at a time (locks/cron overlap).

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: only archive logs whose builds are not running
$log = id(new HarbormasterBuildLogQuery())
  ->setViewer(PhabricatorUser::getOmnipotentUser())
  ->withIDs(array($id))
  ->needBuilds(true)
  ->executeOne();
if ($log->getBuild()->isComplete()) { /* safe to archive */ }

Try / catch

try {
  archiveLog($log, $mode);
} catch (Exception $e) {
  if (preg_match('/hashes differ/', $e->getMessage())) {
    // log was likely still being written: wait for the build to finish and retry once
    waitForBuildCompletion($log);
    archiveLog($log, $mode);
  } else {
    throw $e;
  }
}

Prevention

Pause the daemon (or wait for all builds to finish) before running archive-logs; live logs can change under the rewrite.
Run exactly one archival process at a time; serialize with a lock in cron wrappers.
Investigate every occurrence — a genuine mismatch means corrupted chunks, not a flake; inspect the chunk table for that log ID.

When it happens

Trigger: The log is still being written to (a running build keeps appending chunks) while the archival rewrites it, so before/after hashes cover different data; corrupted or partially missing chunks making reconstruction order-dependent; a storage backend (database blob vs file) returning truncated data under load; concurrent archive-logs runs on the same log.

Common situations: Running archive-logs while the daemon is still executing builds; two overlapping archival runs; disk-full or DB issues truncating chunk writes; an interrupted earlier archival leaving half-plain half-compressed chunks.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/3a8773a7f75adb6f. Report an issue: GitHub.