Duplicate state key %v used by %v and %v. Ensure that state

Error message

Duplicate state key %v used by %v and %v. Ensure that state keys areunique per DoFn

What it means

Beam Go assigns each state field a unique state key derived from the field name. During DoFn validation, if two State structs on the same DoFn resolve to the same state key, this error is thrown, since state would be silently shared/corrupted between the two fields.

Source

Thrown at sdks/go/pkg/beam/core/graph/fn.go:1366

	ps := fn.PipelineState()

	if _, hasSp := fn.methods[processElementName].StateProvider(); hasSp {
		if numIn == MainSingle {
			err := errors.Errorf("ProcessElement uses a StateProvider, but is not keyed")
			return errors.SetTopLevelMsgf(err, "ProcessElement uses a StateProvider, but is not keyed. "+
				"All stateful DoFns must take a key/value pair as an input.")
		}
		if len(ps) == 0 {
			err := errors.Errorf("ProcessElement uses a StateProvider, but noState structs are attached to the DoFn")
			return errors.SetTopLevelMsgf(err, "ProcessElement uses a StateProvider, but no State structs are "+
				"attached to the DoFn. Ensure that you are including the State structs you're using to read/write"+
				"global state as public uppercase member variables.")
		}
		stateKeys := make(map[string]state.PipelineState)
		for _, s := range ps {
			k := s.StateKey()
			if orig, ok := stateKeys[k]; ok {
				err := errors.Errorf("Duplicate state key %v", k)
				return errors.SetTopLevelMsgf(err, "Duplicate state key %v used by %v and %v. Ensure that state keys are"+
					"unique per DoFn", k, orig, s)
			}
			t := s.StateType()
			if t != state.TypeValue && t != state.TypeBag && t != state.TypeCombining && t != state.TypeSet && t != state.TypeMap && t != state.TypeOrderedList {
				err := errors.Errorf("Unrecognized state type %v for state %v", t, s)
				return errors.SetTopLevelMsgf(err, "Unrecognized state type %v for state %v. Currently the only supported state"+
					"types are state.Value, state.Combining, state.Bag, state.Set, state.Map, and state.OrderedList", t, s)
			}
			stateKeys[k] = s
		}
	} else {
		if len(ps) > 0 {
			err := errors.Errorf("ProcessElement doesn't use a StateProvider, but State structs are attached to "+
				"the DoFn: %v", ps)
			return errors.SetTopLevelMsgf(err, "ProcessElement doesn't use a StateProvider, but State structs are "+
				"attached to the DoFn: %v\nEnsure that you are using the StateProvider to perform any reads or writes"+
				"of pipeline state.", ps)

View on GitHub (pinned to 12126d8942)

Solutions

Rename one of the conflicting State fields so each has a unique name/key.
Flatten embedded structs and deduplicate state field names.
Merge the two state fields into one if they intentionally represent the same state.

Example fix

// before
type CountFn struct {
    Seen state.Value[string, int]
    Seen state.Bag[string] // duplicate key
}
// after
type CountFn struct {
    Seen      state.Value[string, int]
    SeenItems state.Bag[string]
}

Defensive patterns

Strategy: validation

Validate before calling

seen := map[string]bool{}
t := reflect.TypeOf(fn)
for i := 0; i < t.NumField(); i++ {
    k := t.Field(i).Name
    if seen[k] { log.Fatalf("duplicate state field name: %s", k) }
    seen[k] = true
}

Try / catch

if err := beam.Run(ctx, p); err != nil {
    if strings.Contains(err.Error(), "Duplicate state key") {
        log.Fatalf("rename one of the colliding State fields: %v", err)
    }
    return err
}

Prevention

Give every State field a distinct, descriptive name
Avoid embedding multiple structs that contain same-named State fields
Search the DoFn struct for repeated field names before building

When it happens

Trigger: Declaring two State fields on one DoFn whose keys collide — typically two fields with the same name in embedded structs, or two State structs constructed with the same state key identifier.

Common situations: Embedding two structs that both contain a State field with the same name; copy-pasting a state field and forgetting to rename it; generated DoFns where template fields weren't uniquified.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/21b1040fd961ac31. Report an issue: GitHub.