apache/beam · error

schema[ ] changed: got fields, want fields

Error message

schema[%v] changed: got %d fields, want %d fields

What it means

Raised while decoding a schema-coded row: the row header read from the stream declares a number of fields that differs from what the decoder expects. This means the encoded data's schema no longer matches the Go struct the coder was built from — the schema changed between encode and decode.

Solutions

1. Regenerate or re-encode the data with the current schema instead of decoding stale data.
2. Add the new field as exported with matching order, or keep schema evolution compatible (append fields at the end) and ensure field count expectations match.
3. Use schema evolution-aware storage or migrate old data with the old coder version.
4. Check that both pipeline stages use the same struct type/version.

Example fix

// before (struct changed incompatibly)
type Row struct { A string }
// reading old data written with two fields

// after (compatible evolution)
type Row struct { A string; B string } // appended in same order as writer's schema

// Before decoding persisted data, compare its header against expectations:
nf, _, err := coder.ReadRowHeader(r)
if err != nil { return err }
if nf != expectedFieldCount {
    return fmt.Errorf("stored schema has %d fields, code expects %d; migrate data first", nf, expectedFieldCount)
}

row, err := decodeRow(data)
if err != nil && strings.Contains(err.Error(), "schema") && strings.Contains(err.Error(), "changed") {
    return migrateAndDecode(data) // decode with old coder version
}

When it happens

Trigger: Calling the returned decode closure (from decoderForStructReflect) on a byte stream whose ReadRowHeader reports nf != len(coder.fields), e.g. decoding an old snapshot/output with a newer or older version of the struct, or mixing pipeline outputs with changed types.

Common situations: Deploying an updated binary that added/removed struct fields and then re-reading materialized data or rerunning with cached/staged input; also cross-version data exchange between jobs.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Could not decode the value from Row
• Could not determine the generic type of the list
• error decoding bool field
• error decoding double field
• error decoding single byte field

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/181f1305068c696b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/graph/coder/row_decoder.go:181

			// Add a no-op coder to fill in field index
			coder.fields = append(coder.fields, typeDecoderFieldReflect{decode: func(rv reflect.Value, r io.Reader) error {
				return nil
			}})
			continue
		}
		dec, err := b.decoderForSingleTypeReflect(sf.Type)
		if err != nil {
			return nil, err
		}
		coder.fields = append(coder.fields, dec)
	}
	return func(rv reflect.Value, r io.Reader) error {
		nf, nils, err := ReadRowHeader(r)
		if err != nil {
			return err
		}
		if nf != len(coder.fields) {
			return errors.Errorf("schema[%v] changed: got %d fields, want %d fields", coder.typ, nf, len(coder.fields))
		}
		for i, f := range coder.fields {
			if IsFieldNil(nils, i) {
				continue
			}
			fv := rv.Field(i)
			if f.addr {
				fv = fv.Addr()
			}
			if err := f.decode(fv, r); err != nil {
				return err
			}
		}
		return nil
	}, nil
}

func reflectDecodeBool(rv reflect.Value, r io.Reader) error {

View on GitHub (pinned to 12126d8942)