{"record":{"id":"ad5b9ccf14f33264","repo":"hashicorp/nomad","slug":"failed-to-get-destination-log-d-w","errorCode":null,"errorMessage":"failed to get destination log %d: %w","messagePattern":"failed to get destination log (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"helper/raftutil/migrate.go","lineNumber":333,"sourceCode":"\n\t// If we have logs, spot-check a few entries for data integrity.\n\tif srcFirst > 0 && srcLast > 0 {\n\t\tindicesToCheck := []uint64{srcFirst}\n\t\tif srcLast > srcFirst {\n\t\t\t// Check middle entry.\n\t\t\tmiddle := srcFirst + (srcLast-srcFirst)/2\n\t\t\tindicesToCheck = append(indicesToCheck, middle)\n\t\t\t// Check last entry.\n\t\t\tindicesToCheck = append(indicesToCheck, srcLast)\n\t\t}\n\n\t\tfor _, idx := range indicesToCheck {\n\t\t\tvar srcLog, dstLog raft.Log\n\t\t\tif err := src.GetLog(idx, &srcLog); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get source log %d: %w\", idx, err)\n\t\t\t}\n\t\t\tif err := dst.GetLog(idx, &dstLog); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get destination log %d: %w\", idx, err)\n\t\t\t}\n\n\t\t\tif srcLog.Index != dstLog.Index || srcLog.Term != dstLog.Term ||\n\t\t\t\tsrcLog.Type != dstLog.Type || string(srcLog.Data) != string(dstLog.Data) {\n\t\t\t\treturn fmt.Errorf(\"log entry %d mismatch\", idx)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":315,"sourceCodeEnd":345,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L315-L345","documentation":"During the spot-check phase of verifyMigration (migrate.go:333), reading a log entry at a sampled index (first, middle, or last) from the DESTINATION LogStore via GetLog failed. The migrated destination must return every entry the source can; a read failure means incomplete or unreadable data was written, so the migration aborts with the index and wrapped cause.","triggerScenarios":"dst.GetLog(idx, &dstLog) errors for a sampled idx after copy — the copy loop skipped or partially wrote entries, the destination bolt transaction was never committed/flushed, or the destination file is corrupt or on a full disk.","commonSituations":"Destination disk full or permission denied during migration; process killed mid-migration leaving an unsynced bolt DB; destination implementation returning errors for entries never inserted due to a copy-loop bug.","solutions":["Check the wrapped error for the underlying destination failure (disk full, permissions, corruption).","Verify MigrateToWAL copied every entry from srcFirst..srcLast and committed its transaction before verification.","Re-run the migration on a fresh destination path to rule out a partially written DB.","Close/flush the destination store cleanly after copy, then reopen it for verification.","Check destination disk space and file permissions before re-running."],"exampleFix":"// before\nraftutil.MigrateToWAL(src, dstPath)\nverifyMigration(src, dst) // dst.GetLog fails: copy not flushed\n// after\nif err := raftutil.MigrateToWAL(src, dstPath); err != nil { return err }\ndst.Close() // ensure bolt tx committed and flushed\ndst, _ = raftboltdb.NewBoltStore(dstPath)\nverifyMigration(src, dst)","handlingStrategy":"validation","validationCode":"// Spot-check the destination is readable right after the copy, before full verification:\nfirst, _ := dst.FirstIndex()\nlast, _ := dst.LastIndex()\nvar l raft.Log\nif first > 0 {\n    if err := dst.GetLog(last, &l); err != nil {\n        return fmt.Errorf(\"destination unreadable at %d after copy; disk/commit issue: %w\", last, err)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := raftutil.MigrateToWAL(src, dstPath); err != nil {\n    if strings.Contains(err.Error(), \"failed to get destination log\") {\n        log.Printf(\"destination copy incomplete: %v (cause: %v)\", err, errors.Unwrap(err))\n        os.Remove(dstPath) // drop the bad copy and retry fresh\n    }\n    return err\n}","preventionTips":["Ensure disk has enough free space for the full log before migrating.","Close/sync the destination bolt store cleanly after the copy, then reopen for verification.","Never kill the process mid-migration; run it under a supervisor with completion checks.","Verify the copy loop iterates srcFirst..srcLast inclusively with no skipped indices.","Check destination file permissions are writable by the migrating process."],"tags":["raft","logstore","migration","data-integrity"],"backgroundTag":"raft-log-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}