What causes "batch: failed to marshal worker UUID: %v" in apache/beam?

Inspect the error message; if uuid.New() returned an invalid UUID, check for tampered/incorrect use of the uuid package.

apache/beam · error

batch: failed to marshal worker UUID

Error message

batch: failed to marshal worker UUID: %v

What it means

makeShardID lazily generates a per-worker UUID via uuid.New().MarshalBinary() and panics if marshaling the UUID bytes fails. Google's UUID MarshalBinary virtually never fails (UUIDs are fixed 16 bytes), so this panic represents an unexpected internal invariant break while constructing the 24-byte shard ID.

Solutions

Inspect the error message; if uuid.New() returned an invalid UUID, check for tampered/incorrect use of the uuid package.
Ensure the google/uuid dependency is intact and at a normal version (go mod tidy / go mod download).
Regenerate go.sum/vendor if the module cache is corrupted.
As a last resort, replace with a known-good UUID construction (uuid.Must(uuid.NewRandom())) in a fork.

Defensive patterns

Strategy: try-catch

Try / catch

defer func() {
	if r := recover(); r != nil {
		log.Fatalf("shard ID init failed: %v", r) // init-time failure is fatal
	}
}()

Prevention

Keep the google/uuid module dependency healthy (go mod verify)
Avoid vendoring with a corrupted module cache
Treat UUID marshal failures as a fatal environment problem

When it happens

Trigger: First call to makeShardID in the worker process (guarded by workerUUIDOnce) where uuid.New().MarshalBinary() returns a non-nil error; also reachable through shard-ID generation from batch element processing.

Common situations: See trigger scenarios.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2ba202581fc82b1e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/transforms/batch/batch.go:509

) {
	emit(ShardedKey[K]{Key: key, ShardID: makeShardID()}, value)
}

var (
	workerUUIDOnce sync.Once
	workerUUIDVal  [16]byte
	shardCounter   atomic.Uint64
)

// makeShardID returns a 24-byte shard identifier: a 16-byte worker
// UUID fixed per process plus an 8-byte atomic counter, big-endian.
// The layout mirrors the Java and Python shapes exactly so the wire
// bytes of cross-language round-trips remain aligned.
func makeShardID() []byte {
	workerUUIDOnce.Do(func() {
		b, err := uuid.New().MarshalBinary()
		if err != nil {
			panic(fmt.Sprintf("batch: failed to marshal worker UUID: %v", err))
		}
		copy(workerUUIDVal[:], b)
	})
	out := make([]byte, 24)
	copy(out[:16], workerUUIDVal[:])
	counter := shardCounter.Add(1)
	binary.BigEndian.PutUint64(out[16:24], counter)
	return out
}

// writeVarInt writes a varint-encoded int64 to buf (unsigned,
// little-endian base-128).
func writeVarInt(buf *bytes.Buffer, v int64) {
	u := uint64(v)
	for u >= 0x80 {
		buf.WriteByte(byte(u) | 0x80)
		u >>= 7
	}

View on GitHub (pinned to 12126d8942)