kopia/kopia · error

new set

Error message

new set

What it means

IterateUnreferencedPacks starts by allocating a bigmap.Set to track used pack IDs; if bigmap.NewSet fails, the error is wrapped as 'new set'. bigmap sets spill to temporary storage/memory, so this almost always indicates a failure allocating memory or creating the temp spill files.

Solutions

1. Free memory or raise the container/host memory limit
2. Ensure the temp directory used for spilling exists and is writable (check TMPDIR)
3. Reduce concurrent workloads sharing memory with the iteration
4. Retry the operation once memory pressure subsides

Example fix

// before
cmd := exec.Command(ctx, "kopia", "blob", "list") // inside a 64MB container
// after
// raise memory limit for maintenance jobs
docker run -m 512m ... kopia maintenance run

if debug.FreeMemory() < 256<<20 { return errors.New("insufficient memory for pack set allocation") }
if f, err := os.CreateTemp("", "bigmap-*"); err != nil { return err } else { f.Close(); os.Remove(f.Name()) }

usedPacks, err := bigmap.NewSet(ctx)
if err != nil {
    return fmt.Errorf("cannot allocate pack set (memory/temp dir?): %w", err)
}

When it happens

Trigger: Calling WriteManager.IterateUnreferencedPacks when bigmap.NewSet cannot initialize — out of memory, or temp directory unavailable/unwritable.

Common situations: Running on memory-constrained hosts or containers with low memory limits, or an environment where the temp directory (TMPDIR) is read-only or full.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• NewSet
• unable to map region
• ACL entry for a given user and target already exists
• BLOB not found
• cache directory was not absolute, refusing to delete

AI-assisted analysis of kopia/kopia@82495e54b5 (2026-09-07). Data as JSON: /api/errors/29e4f815f24c3a50. Report an issue: GitHub.

Appendix: source

Thrown at repo/content/content_manager_iterate.go:235

			return nil
		}); err != nil {
		return errors.Wrap(err, "error iterating contents")
	}

	for _, v := range packUsage {
		if err := callback(*v); err != nil {
			return err
		}
	}

	return nil
}

// IterateUnreferencedPacks returns the list of unreferenced storage blobs.
func (bm *WriteManager) IterateUnreferencedPacks(ctx context.Context, blobPrefixes []blob.ID, parallelism int, callback func(blob.Metadata) error) error {
	usedPacks, err := bigmap.NewSet(ctx)
	if err != nil {
		return errors.Wrap(err, "new set")
	}

	defer usedPacks.Close(ctx)

	contentlog.Log(ctx, bm.log, "determining blobs in use")
	// find packs in use
	if err := bm.IteratePacks(
		ctx,
		IteratePackOptions{
			Prefixes:                           blobPrefixes,
			IncludePacksWithOnlyDeletedContent: true,
		},
		func(pi PackInfo) error {
			if pi.ContentCount > 0 {
				usedPacks.Put(ctx, []byte(pi.PackID))
			}

			return nil

View on GitHub (pinned to 82495e54b5)