apache/beam · error

Could not unmarshal SourceConfig: %v

Error message

Could not unmarshal SourceConfig: %v

What it means

SourceConfigBuilder.BuildFromJSON() panics when the provided JSON cannot be decoded into SourceConfig. The decoder uses DisallowUnknownFields(), so both malformed JSON and any unrecognized field name fail decoding. This is an early-validation guard so bad pipeline configs abort at graph construction instead of at runtime.

Source

Thrown at sdks/go/pkg/beam/io/synthetic/source.go:294

// BuildFromJSON constructs the SourceConfig by populating it with the parsed
// JSON. Panics if there is an error in the syntax of the JSON or if the input
// contains unknown object keys.
//
// An example of valid JSON object:
//
//	{
//		 "num_records": 5,
//		 "key_size": 5,
//		 "value_size": 5,
//		 "num_hot_keys": 5,
//	}
func (b *SourceConfigBuilder) BuildFromJSON(jsonData []byte) SourceConfig {
	decoder := json.NewDecoder(bytes.NewReader(jsonData))
	decoder.DisallowUnknownFields()

	if err := decoder.Decode(&b.cfg); err != nil {
		panic(fmt.Sprintf("Could not unmarshal SourceConfig: %v", err))
	}
	return b.cfg
}

// SourceConfig is a struct containing all the configuration options for a
// synthetic source. It should be created via a SourceConfigBuilder, not by
// directly initializing it (the fields are public to allow encoding).
type SourceConfig struct {
	NumElements    int64   `json:"num_records" beam:"num_records"`
	InitialSplits  int64   `json:"initial_splits" beam:"initial_splits"`
	KeySize        int64   `json:"key_size" beam:"key_size"`
	ValueSize      int64   `json:"value_size" beam:"value_size"`
	NumHotKeys     int64   `json:"num_hot_keys" beam:"num_hot_keys"`
	HotKeyFraction float64 `json:"hot_key_fraction" beam:"hot_key_fraction"`
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Validate the JSON with a plain json.Unmarshal into SourceConfig first and inspect the error message for the offending field.
2. Remove or rename unknown fields so they exactly match SourceConfig field names (case-sensitive, no snake_case).
3. Check value types: durations must be strings like '10ms', counts must be numbers.
4. Verify the Beam SDK version supports the fields you are passing.

Example fix

// before
{"NumHotKeys": "10"}
// after
{"NumHotKeys": 10}

var cfg synthetic.SourceConfig
if err := json.Unmarshal(jsonData, &cfg); err != nil {
    return fmt.Errorf("invalid SourceConfig JSON: %w", err)
}

When it happens

Trigger: Passing JSON with a typo'd or unknown field to BuildFromJSON (e.g. "numHotkeys" vs "NumHotKeys"), malformed JSON syntax, or a value whose type does not match the struct field (string where a number is expected).

Common situations: Hand-authoring synthetic source options in pipeline templates or YAML-embedded JSON; version drift where a newer/older Beam expects fields the JSON doesn't match; case-sensitivity mistakes in field names.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• encoding partition function
• unmarshalling partitionFn data
• trigger.AfterCount(%v) must be a positive integer
• can't apply processing delay of less than a millisecond. Got
• can't apply an alignment period of less than a millisecond.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8d1c2706ffc1062c. Report an issue: GitHub.