ipfs/kubo · error

iterating orphaned keystore data: %w

Error message

iterating orphaned keystore data: %w

What it means

While iterating results from the orphaned-keystore query via results.Next(), a per-entry error on a result (result.Error) is wrapped as "iterating orphaned keystore data: %w". Unlike error 598, the query started successfully; failure happened partway through the result stream. Entries already deleted in the open batch may or may not have been committed (the batch is not yet committed at this point), so the purge is effectively partial/no-op and safe to rerun.

Source

Thrown at core/node/provider.go:563

	syncKey := datastore.NewKey(orphanedPrefix)

	results, err := ds.Query(ctx, query.Query{
		Prefix:   orphanedPrefix,
		KeysOnly: true,
	})
	if err != nil {
		return fmt.Errorf("querying orphaned keystore data: %w", err)
	}
	defer results.Close()

	var batch datastore.Batch
	var count, pending int
	for result := range results.Next() {
		if ctx.Err() != nil {
			return ctx.Err()
		}
		if result.Error != nil {
			return fmt.Errorf("iterating orphaned keystore data: %w", result.Error)
		}
		if batch == nil {
			batch, err = ds.Batch(ctx)
			if err != nil {
				return fmt.Errorf("creating batch for orphaned keystore cleanup: %w", err)
			}
		}
		if err := batch.Delete(ctx, datastore.NewKey(result.Key)); err != nil {
			return fmt.Errorf("batch deleting orphaned key %s: %w", result.Key, err)
		}
		count++
		pending++
		if pending >= purgeBatchSize {
			if err := batch.Commit(ctx); err != nil {
				return fmt.Errorf("committing orphaned keystore cleanup batch: %w", err)
			}
			if err := ds.Sync(ctx, syncKey); err != nil {
				return fmt.Errorf("syncing orphaned keystore cleanup: %w", err)

View on GitHub (pinned to 329838acdf)

Solutions

1. Read the wrapped cause to identify the specific key/entry that failed to iterate.
2. Check the storage backing the repo for I/O problems (dmesg, SMART, mount health) and disk space.
3. Stop other processes using the same IPFS_PATH and rerun the purge — it is resumable because deletes are batched and committed only after iteration succeeds.
4. Run the datastore's verification/repair tool (flatfs verify, leveldb Compact/repair) on the keystore datastore.
5. If a specific entry is corrupt and unrecoverable, remove that backing file/database segment and rerun.

Example fix

// before
for result := range results.Next() {
    if result.Error != nil {
        return fmt.Errorf("iterating orphaned keystore data: %w", result.Error)
    }
    ...
}
// after (log and skip bad entries, keep sweep resumable)
for result := range results.Next() {
    if result.Error != nil {
        log.Warnw("skipping unreadable orphaned keystore entry", "error", result.Error)
        continue
    }
    ...
}

// Check context before starting so cancellation is not misread as a result error:
if err := ctx.Err(); err != nil {
    return err
}
if err := pingDatastore(ctx, ds); err != nil {
    return fmt.Errorf("datastore not readable, aborting orphan sweep: %w", err)
}

for result := range results.Next() {
    if ctx.Err() != nil {
        batchDiscard(batch) // do not commit partial deletes
        return ctx.Err()
    }
    if result.Error != nil {
        batchDiscard(batch)
        if isTransient(result.Error) {
            return retryPurge(ctx, ds, 1)
        }
        return fmt.Errorf("iterating orphaned keystore data: %w", result.Error)
    }
    // ... normal handling
}

When it happens

Trigger: purgeOrphanedKeystoreData enumerating a result set where one or more results carry result.Error — typically an I/O error reading a key from the underlying keystore datastore mid-iteration, or a context cancellation surfaced through the result stream (ctx.Err() is checked first, so this fires for genuine result errors).

Common situations: Failing disk sectors or NFS/network-attached repo storage dropping reads mid-scan; keystore files removed by another process during the sweep; corrupted entries in flatfs/levelds that error on read; concurrent datastore access from two processes on the same IPFS_PATH.

Related errors

• failed to read key: %w
• cannot list keys in keystore: %w
• unexpected keystore suffix %q, expected "0" or "1"
• creating datastore config for %s: %w
• querying orphaned keystore data: %w

AI-assisted analysis of ipfs/kubo@329838acdf (2026-09-03). Data as JSON: /api/errors/9001a41608c4e180. Report an issue: GitHub.