apache/beam · error

shardedKeyEncoder: Elm2 must be []byte shardID (got %T)

Error message

shardedKeyEncoder: Elm2 must be []byte shardID (got %T)

What it means

The shardedKeyEncoder encodes a sharded key as encoded bytes: the shard ID (Elm2) followed by the key (Elm). It requires Elm2 to be a []byte shard ID. If the FullValue's Elm2 holds any other type, encoding cannot proceed and this error is returned instead of silently mis-encoding.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/coder.go:1387

// shardedKeyEncoder encodes ShardedKey-typed values in the standard
// beam:coder:sharded_key:v1 wire format:
//
//	ByteArrayCoder.encode(ShardID) ++ keyCoder.encode(Key)
//
// Runtime values are carried by a FullValue whose Elm holds the user key
// and whose Elm2 holds the []byte shard identifier — the same two-part
// convention used by the KV coder. This matches the Java
// util.ShardedKey.Coder and Python sharded_key encodings exactly; any
// divergence of a single byte would silently corrupt cross-SDK pipelines.
type shardedKeyEncoder struct {
	key ElementEncoder
}

func (e *shardedKeyEncoder) Encode(val *FullValue, w io.Writer) error {
	shardID, ok := val.Elm2.([]byte)
	if !ok {
		return errors.Errorf(
			"shardedKeyEncoder: Elm2 must be []byte shardID (got %T)", val.Elm2)
	}
	if err := coder.EncodeBytes(shardID, w); err != nil {
		return errors.WithContext(err, "shardedKeyEncoder: shardID")
	}
	return e.key.Encode(&FullValue{Elm: val.Elm}, w)
}

// shardedKeyDecoder is the inverse of shardedKeyEncoder. Decoded values
// are placed in FullValue{Elm: key, Elm2: shardID}.
type shardedKeyDecoder struct {
	key ElementDecoder
}

func (d *shardedKeyDecoder) DecodeTo(r io.Reader, fv *FullValue) error {
	shardID, err := coder.DecodeBytes(r)
	if err != nil {
		return errors.WithContext(err, "shardedKeyDecoder: shardID")

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure the FullValue passed to a sharded-key coder has Elm2 set to a []byte shard ID.
2. If Elm2 is nil, assign an explicit shard ID: fv.Elm2 = []byte("shard-0").
3. Verify the coder used matches the element type; a plain KV coder should be used when no shard ID exists.

Example fix

// before
fv := &FullValue{Elm: key}
enc.Encode(fv, w) // Elm2 is nil

// after
fv := &FullValue{Elm: key, Elm2: []byte(shardID)}
enc.Encode(fv, w)

if fv.Elm2 == nil { fv.Elm2 = []byte(defaultShardID) }
shard, ok := fv.Elm2.([]byte)
if !ok { return fmt.Errorf("Elm2 must be []byte, got %T", fv.Elm2) }

func hasShardID(fv *FullValue) bool { _, ok := fv.Elm2.([]byte); return ok }

if err := enc.Encode(fv, w); err != nil {
	if strings.Contains(err.Error(), "shardedKeyEncoder: Elm2 must be []byte") {
		// fix element or fall back to plain KV coder
	}
	return err
}

When it happens

Trigger: Calling Encode on a sharded-key coder (created via NewShardedKeyEncoder) with a FullValue whose Elm2 is nil or a non-[]byte value, e.g. Elm2 left unset or set to a string/int shard ID.

Common situations: Custom exec units or test harnesses constructing FullValue objects by hand for sharded keys; pipeline construction bugs where the sharded key coder is attached but the plan emits plain KVs without shard bytes.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• type must be bigtableio.Mutation but is: %v
• invalid pos type: %T
• encoding %v, expected: %q
• received unknown value type: want []byte, got %T
• unknown fieldtype: %T

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/690009542654f1da. Report an issue: GitHub.