What causes "base64 decode for %v failed" in apache/beam?

Regenerate the pipeline serialization instead of editing the payload by hand.

base64 decode for failed

Error message

base64 decode for %v failed

What it means

decodeDataflowCustomCoder decodes the base64-encoded payload of a custom coder reference into a v1pb.CustomCoder proto. This error wraps a failure of protox.DecodeBase64, meaning the payload string is not valid base64 or not a valid protobuf encoding of CustomCoder.

Solutions

Regenerate the pipeline serialization instead of editing the payload by hand.
Verify the payload is standard (not URL-safe/unpadded variant mismatch) base64 of a CustomCoder proto.
Check that the pipeline spec wasn't transformed by a layer that escapes or rewrites strings (e.g. HTML templates, YAML).
Align SDK versions on both sides of job submission.

Example fix

// before: hand-crafted payload
ref := &CoderRef{Payload: "<pasted string>"}
// after: round-trip via the library
w, err := graphx.EncodeCoderRef(coder)
...

Defensive patterns

Strategy: validation

Validate before calling

if _, err := base64.StdEncoding.DecodeString(payload); err != nil {
    return fmt.Errorf("custom coder payload is not valid base64: %w", err)
}

Try / catch

if err != nil {
    if strings.Contains(err.Error(), "base64 decode") {
        // regenerate pipeline spec instead of retrying
    }
}

Prevention

Never hand-edit serialized pipeline specs
Avoid transport layers that rewrite strings (HTML escaping, line wrapping)
Round-trip coder refs through graphx.Encode/Decode in tests

When it happens

Trigger: Calling DecodeCoderRef on a coder ref whose custom-coder payload string was corrupted, truncated, double-encoded, or not produced by the corresponding encodeDataflowCustomCoder.

Common situations: Hand-editing serialized pipeline JSON; transporting pipeline specs through channels that mangle strings (HTML-escaping, newline wrapping); version mismatch where the payload format changed.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9705dbc0a25801df. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/graphx/dataflow.go:364

		schm := &pipepb.Schema{}
		if err := protox.DecodeBase64(subC.Type, schm); err != nil {
			return nil, err
		}
		t, err := schema.ToType(schm)
		if err != nil {
			return nil, err
		}
		return &coder.Coder{Kind: coder.Row, T: typex.New(t)}, nil

	default:
		return nil, errors.Errorf("custom coders must be length prefixed: %+v", c)
	}
}

func decodeDataflowCustomCoder(payload string) (*coder.Coder, error) {
	var ref v1pb.CustomCoder
	if err := protox.DecodeBase64(payload, &ref); err != nil {
		return nil, errors.Wrapf(err, "base64 decode for %v failed", payload)
	}
	custom, err := decodeCustomCoder(&ref)
	if err != nil {
		return nil, err
	}
	t := typex.New(custom.Type)
	return &coder.Coder{Kind: coder.Custom, T: t, Custom: custom}, nil
}

func isCoGBKList(ref *CoderRef) ([]*CoderRef, bool) {
	if ref.Type != lengthPrefixType {
		return nil, false
	}
	ref2 := ref.Components[0]
	if ref2.Type != cogbklistType {
		return nil, false
	}
	return ref2.Components, true

View on GitHub (pinned to 12126d8942)