{"record":{"id":"bbe8434fe487fedf","repo":"ipfs/kubo","slug":"embedded-migration-phase-failed-w","errorCode":null,"errorMessage":"embedded migration phase failed: %w","messagePattern":"embedded migration phase failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"repo/fsrepo/migrations/migrations.go","lineNumber":459,"sourceCode":"\t\t\t}\n\n\t\t\t// Legacy migrations only support HTTPS downloads\n\t\t\tfetcher, err := GetMigrationFetcher(migrationCfg.DownloadSources, GetDistPathEnv(CurrentIpfsDist), nil)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get migration fetcher: %w\", err)\n\t\t\t}\n\t\t\tdefer fetcher.Close()\n\n\t\t\tif err = RunMigration(ctx, fetcher, embeddedMigrationsMinVersion, ipfsDir, allowDowngrade); err != nil {\n\t\t\t\treturn fmt.Errorf(\"external migration phase failed: %w\", err)\n\t\t\t}\n\t\t}\n\n\t\t// Phase 2: Use embedded migrations for v16+\n\t\tlogger.Printf(\"Phase 2: Embedded migration from v%d to v%d\", embeddedMigrationsMinVersion, targetVer)\n\t\terr = RunEmbeddedMigrations(ctx, targetVer, ipfsDir, allowDowngrade)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"embedded migration phase failed: %w\", err)\n\t\t}\n\n\t\tlogger.Printf(\"Hybrid migration completed successfully: v%d → v%d\", currentVer, targetVer)\n\t\treturn nil\n\t}\n\n\t// Case 4: Reverse hybrid migration (≥16 to <16)\n\t// Use embedded migrations for ≥16 steps, then external migrations for <16 steps\n\tlogger.Printf(\"Starting reverse hybrid migration from version %d to %d\", currentVer, targetVer)\n\tlogger.Print(\"Using reverse hybrid migration strategy: embedded to v16, then external\")\n\n\t// Phase 1: Use embedded migrations from current version down to v16 (if needed)\n\tif currentVer > embeddedMigrationsMinVersion {\n\t\tlogger.Printf(\"Phase 1: Embedded downgrade from v%d to v%d\", currentVer, embeddedMigrationsMinVersion)\n\t\terr = RunEmbeddedMigrations(ctx, embeddedMigrationsMinVersion, ipfsDir, allowDowngrade)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"embedded downgrade phase failed: %w\", err)\n\t\t}","sourceCodeStart":441,"sourceCodeEnd":477,"githubUrl":"https://github.com/ipfs/kubo/blob/329838acdfafae224582930457efe80aa217afc0/repo/fsrepo/migrations/migrations.go#L441-L477","documentation":"After the external phase completes, RunHybridMigrations runs phase 2: embedded (compiled-in) migrations from v16 up to the target version via RunEmbeddedMigrations. If any embedded migration step fails, the error is wrapped as \"embedded migration phase failed\". Unlike the external phase this requires no downloads, so failures almost always mean repo state, disk, or context cancellation problems.","triggerScenarios":"RunHybridMigrations upgrade path reaching RunEmbeddedMigrations(ctx, targetVer, ipfsDir, allowDowngrade) after external migrations succeed, and an embedded migration returns an error: datastore open/write failure, a migration-specific data error, disk full, or ctx cancelled mid-run.","commonSituations":"Long `ipfs daemon` startup on a large repo killed by Ctrl-C or systemd timeout mid-migration; disk full while rewriting the datastore; repo partially migrated by an earlier interrupted attempt; datastore engine misconfigured in config after upgrade.","solutions":["Re-run the migration — each version step is idempotent and resumes from the recorded repo version; check <ipfsDir>/version to see progress","Free disk space (migrations can temporarily need space comparable to the datastore) and retry","Increase the service timeout (e.g. systemd TimeoutStartSec) so long migrations are not killed","Back up the repo, then inspect the failing step's log output; run `ipfs repo fsck` if corruption is suspected"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"ver, err := migrations.RepoVersion(ipfsPath)\nif err != nil {\n\treturn err\n}\nif free, err := freeDisk(ipfsPath); err == nil && free < minRequiredBytes {\n\treturn fmt.Errorf(\"insufficient disk for embedded migration v%d->v%d\", ver, targetVer)\n}\n","typeGuard":"func isEmbeddedMigrationFailure(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"embedded migration phase failed\")\n}\n","tryCatchPattern":"if err := migrations.RunHybridMigrations(ctx, targetVer, ipfsPath, false); err != nil {\n\tif isEmbeddedMigrationFailure(err) && errors.Is(err, context.Canceled) {\n\t\tlog.Print(\"embedded migration was interrupted; it is safe to re-run the daemon to resume\")\n\t\treturn err\n\t}\n\treturn fmt.Errorf(\"embedded migration failed; check disk space and repo integrity: %w\", err)\n}\n","preventionTips":["Never kill the daemon mid-migration; raise service start timeouts","Keep free disk space comparable to datastore size before upgrading","Back up the repo before major upgrades","Re-running is safe: migrations resume from the recorded repo version"],"tags":["go","migrations","embedded","datastore","repo-upgrade"],"backgroundTag":"embedded-migration-failed","analyzedSha":"329838acdfafae224582930457efe80aa217afc0","analyzedAt":"2026-09-03T18:30:52.135Z","contentChangedAt":"2026-09-03T18:30:52.135Z","schemaVersion":2},"datasetVersion":"2026-09-11T00:17:11.886Z"}