error writing extra random suffix to ensure indexes are…

Error message

error writing extra random suffix to ensure indexes are always globally unique

What it means

After generating the random suffix, BuildShards (when stable=false) writes it into the shard's write buffer. If buf.Write fails, partial buffers are closed via closeShards() and the error is wrapped as "error writing extra random suffix to ensure indexes are always globally unique". This signals an in-memory write-buffer failure while appending the uniqueness suffix.

Solutions

Check the wrapped cause for memory/resource exhaustion and reduce concurrent shard builds or batch size.
Ensure closeShards() is not being invoked before this write (buffers must still be open).
Retry the operation once memory pressure is relieved; if uniqueness is unnecessary, use stable=true to skip the suffix write.

Example fix

// before
shards, cleanup, err := builder.BuildShards(v, false, hugeSize) // OOM during suffix write
// after
builder2 := builder.Split() // smaller batches
shards, cleanup, err := builder2.BuildShards(v, false, hugeSize)

Defensive patterns

Strategy: retry

Try / catch

shards, cleanup, err := builder.BuildShards(v, false, size)
if err != nil {
	runtime.GC()
	shards, cleanup, err = builder.BuildShards(v, false, size)
}

Prevention

Bound concurrent shard builds to limit memory pressure in write buffers.
Use stable=true to skip the random-suffix write when uniqueness is handled elsewhere.
Monitor memory before large index compactions.

When it happens

Trigger: Calling BuildShards with stable=false when the gather.WriteBuffer's Write call fails — typically memory allocation failure or a buffer implementation error while appending len(randomSuffix) bytes.

Common situations: Out-of-memory / resource exhaustion while building very large shard sets; a defective or prematurely closed write buffer.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of kopia/kopia@82495e54b5 (2026-09-07). Data as JSON: /api/errors/d69812173cc4cd23. Report an issue: GitHub.

Appendix: source

Thrown at repo/content/index/one_use_index_builder.go:130

		dataShardsBuf = append(dataShardsBuf, buf)

		if err := buildSortedContents(s, buf, indexVersion); err != nil {
			closeShards()

			return nil, nil, errors.Wrap(err, "error building index shard")
		}

		if !stable {
			if _, err := rand.Read(randomSuffix[:]); err != nil {
				closeShards()

				return nil, nil, errors.Wrap(err, "error getting random bytes for suffix")
			}

			if _, err := buf.Write(randomSuffix[:]); err != nil {
				closeShards()

				return nil, nil, errors.Wrap(err, "error writing extra random suffix to ensure indexes are always globally unique")
			}
		}

		dataShards = append(dataShards, buf.Bytes())
	}

	return dataShards, closeShards, nil
}

View on GitHub (pinned to 82495e54b5)