apache/beam · error

number of fields is less than byte array %v < %v

Error message

number of fields is less than byte array %v < %v

What it means

ReadRowHeader decodes a Beam Schema Row header: after reading the field count nf and the nils bit-field byte-array length l, it validates nf >= l because each byte in the nils array can represent at most one field's nil-ness. This error means the header claims more nil-bit bytes than there are fields, so the encoded row is corrupt or was written by an incompatible encoder.

Source

Thrown at sdks/go/pkg/beam/core/graph/coder/row.go:175

// examined once during decoding using the IsFieldNil helper function.
//
// If there are no nil fields encoded,the byte array will be nil, and no
// encoded fields will be nil.
func ReadRowHeader(r io.Reader) (int, []byte, error) {
	nf, err := DecodeVarInt(r) // is for checksum purposes (old vs new versions of a schemas)
	if err != nil {
		return 0, nil, err
	}
	l, err := DecodeVarInt(r) // read the length prefix for the packed bits.
	if err != nil {
		return int(nf), nil, err
	}
	if l == 0 {
		// A zero length byte array means no nils.
		return int(nf), nil, nil
	}
	if nf < l {
		return int(nf), nil, fmt.Errorf("number of fields is less than byte array %v < %v", nf, l)
	}
	nils := make([]byte, l)
	if err := ioutilx.ReadNBufUnsafe(r, nils); err != nil {
		return int(nf), nil, err
	}
	return int(nf), nils, nil
}

// IsFieldNil examines the passed in packed bits nils buffer
// and returns true if the field at that index wasn't encoded
// and can be skipped in decoding.
func IsFieldNil(nils []byte, f int) bool {
	i, b := f/8, f%8
	// https://github.com/apache/beam/issues/21232: The row header can elide trailing 0 bytes,
	// and we shouldn't care if there are trailing 0 bytes when doing a lookup.
	return i < len(nils) && len(nils) != 0 && (nils[i]>>uint8(b))&0x1 == 1
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the same row coder and schema were used on both encode and decode sides.
2. Check for Beam SDK version mismatch between the job that wrote the data and the one reading it.
3. Validate/re-serialize the source data; inspect raw bytes around the header for corruption.
4. Ensure decoded byte offsets are aligned (a prior decode bug can leave the reader mid-record).

Example fix

// before
rowDecoder.Decode(r, typex.NewRowType(wrongSchema, nil)) // schema mismatch
// after
rowDecoder.Decode(r, typex.NewRowType(originalSchema, nil)) // use the schema the data was written with

if l > nf { return fmt.Errorf("corrupt row header: nils bytes %d exceed fields %d", l, nf) }

nf, nils, err := coder.ReadRowHeader(r)
if err != nil && strings.Contains(err.Error(), "number of fields is less than byte array") {
	return fmt.Errorf("data written with incompatible coder/schema: %w", err)
}

When it happens

Trigger: Decoding a stream with row coder ReadRowHeader where the nils-length varint exceeds the preceding field count — typically from data corruption, wrong coder being used to decode, or bytes written by a different Beam version/protocol.

Common situations: Mixing Beam SDK versions between writer and reader; a pipeline deserializing data with the wrong coder assigned; truncated/corrupted serialized data shifting varint boundaries.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• ReadSimpleRowHeader field count: %v, %v
• WriteSimpleRowHeader a 0 length nils bit field: %v
• ReadSimpleRowHeader field count mismatch, got %v, want %v
• varint too long
• error encoding byte: %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/83b755fe7dc0fc14. Report an issue: GitHub.