ipfs/kubo · critical

external migration phase failed: %w

Error message

external migration phase failed: %w

What it means

RunHybridMigrations upgrades an IPFS repo across the boundary of the hybrid scheme: external (downloaded binary) migrations are used below v16 and embedded Go migrations for v16+. When all required external migration binaries are found (or fetched) and runMigrationsFromPath executes them, any failure running those binaries is wrapped as "external migration phase failed". The error carries the underlying cause (binary exit status, download failure, incompatibility).

Source

Thrown at repo/fsrepo/migrations/migrations.go:435

		logger.Printf("Phase 1: External migration from v%d to v%d", currentVer, embeddedMigrationsMinVersion)

		// Check for external migration binaries in PATH first
		migrations, binPaths, err := findMigrations(ctx, currentVer, embeddedMigrationsMinVersion)
		if err != nil {
			return fmt.Errorf("could not determine external migration paths: %w", err)
		}

		foundAll := true
		for _, migName := range migrations {
			if _, exists := binPaths[migName]; !exists {
				foundAll = false
				break
			}
		}

		if foundAll {
			if err = runMigrationsFromPath(ctx, migrations, binPaths, ipfsDir, logger, false); err != nil {
				return fmt.Errorf("external migration phase failed: %w", err)
			}
		} else {
			migrationCfg, err := ReadMigrationConfig(ipfsDir, "")
			if err != nil {
				return fmt.Errorf("could not read migration config: %w", err)
			}

			// Legacy migrations only support HTTPS downloads
			fetcher, err := GetMigrationFetcher(migrationCfg.DownloadSources, GetDistPathEnv(CurrentIpfsDist), nil)
			if err != nil {
				return fmt.Errorf("failed to get migration fetcher: %w", err)
			}
			defer fetcher.Close()

			if err = RunMigration(ctx, fetcher, embeddedMigrationsMinVersion, ipfsDir, allowDowngrade); err != nil {
				return fmt.Errorf("external migration phase failed: %w", err)
			}
		}

View on GitHub (pinned to 329838acdf)

Solutions

1. Re-run the migration (restarting the daemon) after freeing disk space and confirming the repo is intact; migrations are idempotent per version step
2. Run the failing migration manually with `ipfs fs-repo-migrations -to <ver>` or the fs-repo-migrate binary directly to see its full output
3. Delete partially written migration state/binaries under <ipfsDir>/fs-repo-migrations and retry
4. Check the repo with `ipfs repo fsck` / verify version file <ipfsDir>/version is consistent
5. Restore the repo from backup and migrate again if the datastore is corrupt

Example fix

// before (direct call that surfaces the opaque wrapped failure)
err := migrations.RunHybridMigrations(ctx, 16, ipfsPath, false)
// after (pre-flight: check version and run external migrations manually for visibility)
ver, err := migrations.RepoVersion(ipfsPath)
if err == nil && ver < 16 {
	if out, mErr := exec.Command("ipfs", "fs-repo-migrations", "-to", "16").CombinedOutput(); mErr != nil {
		log.Fatalf("external migration failed: %v: %s", mErr, out)
	}
}
err = migrations.RunHybridMigrations(ctx, 16, ipfsPath, false)

ver, err := migrations.RepoVersion(ipfsPath)
if err != nil {
	return fmt.Errorf("cannot read repo version: %w", err)
}
if ver < 16 {
	if fi, err := os.Stat(filepath.Join(ipfsPath, "fs-repo-migrations")); err != nil || !fi.IsDir() {
		return errors.New("pre-v16 repo: ensure fs-repo-migrate binaries are available before upgrading")
	}
}

func isExternalMigrationPhaseFailed(err error) bool {
	return err != nil && strings.Contains(err.Error(), "external migration phase failed")
}

if err := migrations.RunHybridMigrations(ctx, targetVer, ipfsPath, false); err != nil {
	var ctxErr error
	if errors.Is(ctx.Err(), context.Canceled) {
		ctxErr = ctx.Err()
	}
	log.Printf("external migration phase failed: %v (canceled=%v)", err, ctxErr != nil)
	return fmt.Errorf("repo migration interrupted; safe to retry: %w", err)
}

When it happens

Trigger: Calling RunHybridMigrations(ctx, targetVer, ipfsDir, allowDowngrade) on the upgrade path where currentVer < 16 <= targetVer and foundAll==true, and runMigrationsFromPath returns an error: a migration binary fails to execute or exits non-zero, a needed migration binary cannot be run from the resolved path, or the underlying migration process is cancelled via ctx.

Common situations: Daemon startup (`ipfs daemon`) detects an old repo that must pass through the pre-v16 external migrations; the downloaded fs-repo-migrate binary fails on a corrupt or partially-migrated repo; a stale/partial migration binary left in the migrations dir; running out of disk space mid-migration; user Ctrl-C during a long migration.

Related errors

• embedded migration phase failed: %w
• external downgrade phase failed: %w
• cannot get migrations from unknown fetcher type
• abort failed: close: %w, remove: %v
• source field %s does not exist

AI-assisted analysis of ipfs/kubo@329838acdf (2026-09-03). Data as JSON: /api/errors/204835702e67e1ae. Report an issue: GitHub.