{"record":{"id":"e36733b320ce4e1a","repo":"ipfs/kubo","slug":"external-downgrade-phase-failed-w","errorCode":null,"errorMessage":"external downgrade phase failed: %w","messagePattern":"external downgrade phase failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"repo/fsrepo/migrations/migrations.go","lineNumber":500,"sourceCode":"\t\tlogger.Printf(\"Phase 2: External downgrade from v%d to v%d\", embeddedMigrationsMinVersion, targetVer)\n\n\t\t// Check for external migration binaries in PATH first\n\t\tmigrations, binPaths, err := findMigrations(ctx, embeddedMigrationsMinVersion, targetVer)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"could not determine external migration paths: %w\", err)\n\t\t}\n\n\t\tfoundAll := true\n\t\tfor _, migName := range migrations {\n\t\t\tif _, exists := binPaths[migName]; !exists {\n\t\t\t\tfoundAll = false\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\n\t\tif foundAll {\n\t\t\tif err = runMigrationsFromPath(ctx, migrations, binPaths, ipfsDir, logger, true); err != nil {\n\t\t\t\treturn fmt.Errorf(\"external downgrade phase failed: %w\", err)\n\t\t\t}\n\t\t} else {\n\t\t\tmigrationCfg, err := ReadMigrationConfig(ipfsDir, \"\")\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"could not read migration config: %w\", err)\n\t\t\t}\n\n\t\t\t// Legacy migrations only support HTTPS downloads\n\t\t\tfetcher, err := GetMigrationFetcher(migrationCfg.DownloadSources, GetDistPathEnv(CurrentIpfsDist), nil)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get migration fetcher: %w\", err)\n\t\t\t}\n\t\t\tdefer fetcher.Close()\n\n\t\t\tif err = RunMigration(ctx, fetcher, targetVer, ipfsDir, allowDowngrade); err != nil {\n\t\t\t\treturn fmt.Errorf(\"external downgrade phase failed: %w\", err)\n\t\t\t}\n\t\t}","sourceCodeStart":482,"sourceCodeEnd":518,"githubUrl":"https://github.com/ipfs/kubo/blob/329838acdfafae224582930457efe80aa217afc0/repo/fsrepo/migrations/migrations.go#L482-L518","documentation":"On the downgrade path, when all required external migration binaries are found locally, RunHybridMigrations executes them via runMigrationsFromPath with downgrade=true. If any external downgrade migration fails to run or exits non-zero, the error is wrapped as \"external downgrade phase failed\". This is the downgrade counterpart of error 690.","triggerScenarios":"RunHybridMigrations downgrade path (targetVer < 16, allowDowngrade==true) with foundAll==true and runMigrationsFromPath(ctx, migrations, binPaths, ipfsDir, logger, true) returning an error: a fs-repo-migrate-N binary fails to start or exits non-zero while rolling the repo back from v16 to targetVer.","commonSituations":"Rolling a repo back to an older kubo release after a failed upgrade; stale or version-mismatched migration binaries left in <ipfsDir>/fs-repo-migrations; downgrade migration aborts because the newer schema holds data it cannot represent; interrupted earlier downgrade left inconsistent state.","solutions":["Back up the repo, then re-run the downgrade after fixing the underlying cause printed by the wrapped error","Run the failing binary directly (e.g. <ipfsDir>/fs-repo-migrations/fs-repo-migrate-15-14 -path=<ipfsDir>) to see its full output","Delete stale migration binaries from <ipfsDir>/fs-repo-migrations and re-obtain the correct versions","If the newer schema's data is not representable in the older version, export pins/IPNS keys and accept data loss, or keep the newer binary instead"],"exampleFix":"// before (blind retry of failing downgrade)\nerr := migrations.RunHybridMigrations(ctx, 14, ipfsPath, true)\n// after (inspect the failing step manually first)\nout, mErr := exec.Command(filepath.Join(ipfsPath, \"fs-repo-migrations\", \"fs-repo-migrate-15-14\"), \"-path=\"+ipfsPath).CombinedOutput()\nlog.Printf(\"migration output: %s\", out)\nif mErr == nil {\n\terr = migrations.RunHybridMigrations(ctx, 14, ipfsPath, true)\n}","handlingStrategy":"try-catch","validationCode":"for _, bin := range requiredBins {\n\tpath := filepath.Join(ipfsPath, \"fs-repo-migrations\", bin)\n\tif fi, err := os.Stat(path); err != nil || fi.IsDir() {\n\t\treturn fmt.Errorf(\"missing downgrade binary %s\", bin)\n\t}\n\tif err := exec.Command(path, \"--help\").Run(); err != nil {\n\t\treturn fmt.Errorf(\"downgrade binary %s not executable: %w\", bin, err)\n\t}\n}\n","typeGuard":"func isExternalDowngradeFailure(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"external downgrade phase failed\")\n}\n","tryCatchPattern":"if err := migrations.RunHybridMigrations(ctx, targetVer, ipfsPath, true); err != nil {\n\tif isExternalDowngradeFailure(err) {\n\t\tlog.Printf(\"downgrade failed; restore from backup at %s.bak if repo state is inconsistent\", ipfsPath)\n\t\treturn err\n\t}\n\treturn err\n}\n","preventionTips":["Back up the repo before every downgrade","Remove stale migration binaries from <ipfsDir>/fs-repo-migrations before re-running","Run the failing fs-repo-migrate-N binary directly to see raw output","Accept that some newer-schema data cannot be rolled back; export first"],"tags":["go","migrations","downgrade","subprocess"],"backgroundTag":"external-migration-binary-failed","analyzedSha":"329838acdfafae224582930457efe80aa217afc0","analyzedAt":"2026-09-03T18:30:52.135Z","contentChangedAt":"2026-09-03T18:30:52.135Z","schemaVersion":2},"datasetVersion":"2026-09-11T00:17:11.886Z"}