apache/beam · error

err

Error message

err

What it means

Inside batch.RegisterShardedKeyType's ShardedKey encoder closure, encoding the key component via the registered key encoder fails, and the closure panics with the raw error. This is an internal invariant: deterministic encoding of the sharded key should never fail for a properly registered key type.

Solutions

1. Register the key type K with a deterministic coder (RegisterShardedKeyType / beamcoder.RegisterDeterministicCoderWithFuncs) before using GroupIntoBatches.
2. Simplify the key type to encodable primitives/structs.
3. Recover in a wrapper and surface a descriptive error naming the key type.

Example fix

// before
if err := keyEnc.Encode(sk.Key, &buf); err != nil { panic(err) }
// after
if err := keyEnc.Encode(sk.Key, &buf); err != nil {
    panic(fmt.Errorf("batch: encoding ShardedKey key %T: %w", sk.Key, err))
}

// before using GroupIntoBatches, ensure K has a deterministic coder
if !coder.HasDeterministicCoder(reflect.TypeOf(KeyType{})) {
    batch.RegisterShardedKeyType[KeyType]()
}

func keyTypeEncodable[K any]() bool {
    var k K
    switch reflect.TypeOf(k).Kind() {
    case reflect.Chan, reflect.Func, reflect.UnsafePointer:
        return false
    }
    return true
}

defer func() {
    if r := recover(); r != nil {
        log.Fatalf("ShardedKey encode failed (check key type registration): %v", r)
    }
}()

When it happens

Trigger: The enc closure for ShardedKey[K] is invoked during coder serialization when keyEnc.Encode(sk.Key, &buf) returns an error — e.g. the key type K lacks a registered deterministic encoder or contains an unsupported field type.

Common situations: GroupIntoBatches used with a key type that was never registered via RegisterShardedKeyType / beamcoder deterministic coder; keys containing channels, funcs, or unencodable nested types.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• failed encoding key for
• KV coder with more than 2 components
• Nested FullValues must be nested as pointers.
• Runner forgot to LP this Row Coder.
• ShardedKey coder must have only 1 component

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/12a5fed7e3ac0c6f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/transforms/batch/batch.go:112

// Users of other K types must call this at init time.
func RegisterShardedKeyType[K any]() {
	var zero K
	keyT := reflect.TypeOf(zero)
	skT := reflect.TypeOf(ShardedKey[K]{})

	register.DoFn3x0[K, typex.V, func(ShardedKey[K], typex.V)](&wrapShardedKeyFn[K]{})
	register.Emitter2[ShardedKey[K], typex.V]()
	beam.RegisterType(skT)

	keyEnc := beam.NewElementEncoder(keyT)
	keyDec := beam.NewElementDecoder(keyT)

	enc := func(sk ShardedKey[K]) []byte {
		var buf bytes.Buffer
		writeVarInt(&buf, int64(len(sk.ShardID)))
		buf.Write(sk.ShardID)
		if err := keyEnc.Encode(sk.Key, &buf); err != nil {
			panic(err)
		}
		return buf.Bytes()
	}
	dec := func(b []byte) ShardedKey[K] {
		r := bytes.NewReader(b)
		n := readVarInt(r)
		shardID := make([]byte, n)
		if n > 0 {
			if _, err := r.Read(shardID); err != nil {
				panic(err)
			}
		}
		k, err := keyDec.Decode(r)
		if err != nil {
			panic(err)
		}
		return ShardedKey[K]{Key: k.(K), ShardID: shardID}
	}

View on GitHub (pinned to 12126d8942)