apache/beam · error

GroupIntoBatches: key coder for type %v is not deterministic

Error message

GroupIntoBatches: key coder for type %v is not deterministic. Register a deterministic custom coder with coder.RegisterDeterministicCoder, or use a deterministic key type (string, []byte, bool, integer, float)

What it means

GroupIntoBatches shuffles and buffers by key, which requires a deterministic key coder so encoded keys can be compared/decoded consistently across workers. The default coder for the key type is non-deterministic, so the transform panics at build time. Register a deterministic coder or switch key types.

Source

Thrown at sdks/go/pkg/beam/transforms/batch/batch.go:591

// non-deterministic key coders would corrupt state keying. Panics at
// pipeline build time on invalid params, non-KV input, zero limits, or
// a non-deterministic key coder.
func GroupIntoBatches(s beam.Scope, params Params, col beam.PCollection) beam.PCollection {
	s = s.Scope("batch.GroupIntoBatches")

	if err := params.validate(); err != nil {
		panic(fmt.Errorf("GroupIntoBatches: %w", err))
	}
	if !typex.IsKV(col.Type()) {
		panic(fmt.Errorf(
			"GroupIntoBatches: input PCollection must be KV-typed; got %v", col.Type()))
	}

	keyFT := col.Type().Components()[0]
	valFT := col.Type().Components()[1]

	if !beam.NewCoder(keyFT).IsDeterministic() {
		panic(fmt.Errorf(
			"GroupIntoBatches: key coder for type %v is not deterministic. "+
				"Register a deterministic custom coder with "+
				"coder.RegisterDeterministicCoder, or use a deterministic key "+
				"type (string, []byte, bool, integer, float)", keyFT.Type()))
	}

	sizerKind := sizerNone
	if params.BatchSizeBytes > 0 {
		if !isBuiltinSizeable(valFT.Type()) {
			panic(fmt.Errorf(
				"GroupIntoBatches: BatchSizeBytes > 0 requires value type %v "+
					"to be a built-in primitive ([]byte, string, numeric, bool)",
				valFT.Type()))
		}
		sizerKind = sizerPrimitive
	}

	allowedLatenessMs := int64(col.WindowingStrategy().AllowedLateness)

View on GitHub (pinned to 12126d8942)

Solutions

1. Implement and register a deterministic coder: coder.RegisterDeterministicCoder(MyKey{}) in init()
2. Switch keys to inherently deterministic types (string, []byte, bool, numerics, float)
3. Serialize the key to bytes/string yourself before grouping
4. Verify with beam.NewCoder(keyType).IsDeterministic() in a test

Example fix

// before
type Key struct{ A, B string } // non-deterministic default coder
// after
func init() {
    coder.RegisterDeterministicCoder(Key{})
}
type Key struct{ A, B string }

if !beam.NewCoder(reflect.TypeOf(Key{})).IsDeterministic() {
    coder.RegisterDeterministicCoder(Key{})
}

When it happens

Trigger: Calling GroupIntoBatches with a key type where beam.NewCoder(keyFT).IsDeterministic() is false — e.g. struct/map keys without a registered deterministic coder.

Common situations: Using custom struct keys in Go Beam, where the default coder is not deterministic; forgetting coder.RegisterDeterministicCoder in init().

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Expected that the coder is deterministic
• the keyCoder of a GroupByEncryptedKey must be deterministic
• the keyCoder of a GroupByKey must be deterministic
• Window coders must be deterministic.
• the secondary key coder of SortValues must be deterministic

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/75133a6954b5eee1. Report an issue: GitHub.