apache/beam · error

GroupIntoBatches: BatchSizeBytes > 0 requires value type

Error message

GroupIntoBatches: BatchSizeBytes > 0 requires value type %v to be a built-in primitive ([]byte, string, numeric, bool)

What it means

Byte-based batching (BatchSizeBytes > 0) needs each element's size, which the transform can only compute for built-in primitive value types via isBuiltinSizeable. Custom/complex value types have no sizer, so the transform panics. Use element-count batching or a primitive value type.

Solutions

1. Switch to BatchSize (count-based) instead of BatchSizeBytes
2. Change the value type to []byte, string, numeric, or bool
3. Pre-encode values to []byte before batching to enable byte-based sizing
4. Check isBuiltinSizeable-equivalent before setting BatchSizeBytes

Example fix

// before
params := batch.Params{BatchSizeBytes: 1 << 20} // values are custom structs
// after
params := batch.Params{BatchSize: 500} // or emit []byte values

if params.BatchSizeBytes > 0 && !isPrimitiveValueType(reflect.TypeOf(MyVal{})) {
    return errors.New("use BatchSize or primitive value types")
}

When it happens

Trigger: Setting Params.BatchSizeBytes > 0 while the KV value type is a custom struct, slice of structs, or any type not covered by isBuiltinSizeable.

Common situations: Trying to cap batch bytes for protobuf messages or application structs; copying count-limited examples and only changing the limit field.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• capacity of cache cannot be negative, got
• could not unmarshal iterable coder from
• could not unmarshal nullable coder from
• could not unmarshal sharded_key coder from
• empty pipeline

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/abf66161148884e6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/transforms/batch/batch.go:601

		panic(fmt.Errorf(
			"GroupIntoBatches: input PCollection must be KV-typed; got %v", col.Type()))
	}

	keyFT := col.Type().Components()[0]
	valFT := col.Type().Components()[1]

	if !beam.NewCoder(keyFT).IsDeterministic() {
		panic(fmt.Errorf(
			"GroupIntoBatches: key coder for type %v is not deterministic. "+
				"Register a deterministic custom coder with "+
				"coder.RegisterDeterministicCoder, or use a deterministic key "+
				"type (string, []byte, bool, integer, float)", keyFT.Type()))
	}

	sizerKind := sizerNone
	if params.BatchSizeBytes > 0 {
		if !isBuiltinSizeable(valFT.Type()) {
			panic(fmt.Errorf(
				"GroupIntoBatches: BatchSizeBytes > 0 requires value type %v "+
					"to be a built-in primitive ([]byte, string, numeric, bool)",
				valFT.Type()))
		}
		sizerKind = sizerPrimitive
	}

	allowedLatenessMs := int64(col.WindowingStrategy().AllowedLateness)
	valueType := beam.EncodedType{T: valFT.Type()}

	if params.MaxBufferingDuration > 0 {
		fn := &groupIntoBatchesBufferedFn{
			Buffer:            state.MakeBagState[[]byte]("batchBuffer"),
			Count:             state.MakeValueState[int64]("batchCount"),
			ByteSize:          state.MakeValueState[int64]("batchBytes"),
			TimerSet:          state.MakeValueState[bool]("batchTimerSet"),
			Buffering:         timers.InProcessingTime("batchBuffering"),
			WindowEnd:         timers.InEventTime("batchWindowEnd"),

View on GitHub (pinned to 12126d8942)