apache/beam · error

failed to matched a field for SQL column

Error message

failed to matched a %v field for SQL column: %v

What it means

databaseio maps SQL result columns to fields of your Go record struct via reflection (mapFields in util.go). It tries three matching strategies: exact name, case-insensitive name, and case-insensitive name with underscores stripped; an optional `column:"..."` struct tag is also consulted. When a returned column cannot be matched to any exported field of the struct, this error is thrown so the mismatch surfaces immediately instead of silently dropping data.

Solutions

1. Add or update the struct field so its name (or case/underscore-insensitive variant) matches the SQL column exactly
2. Add an explicit struct tag: `column:"the_sql_column_name"` on the matching field
3. Narrow the query to select only columns the struct actually has (avoid SELECT *)
4. Export the intended field — unexported fields are ignored by mapFields
5. Log/inspect the actual column list (e.g. via rows.Columns()) and compare to the struct fields

Example fix

// before
type User struct {
    id    int    // unexported: never matched
    Email string
}
// after
type User struct {
    ID    int    `column:"user_id"`
    Email string
}

func validateColumnsMatchStruct(columns []string, t reflect.Type) error {
    for _, c := range columns {
        found := false
        for i := 0; i < t.NumField(); i++ {
            name := t.Field(i).Name
            tag := t.Field(i).Tag.Get("column")
            norm := strings.ReplaceAll(strings.ToLower(c), "_", "")
            if tag == c || name == c || strings.EqualFold(name, c) || strings.EqualFold(strings.ReplaceAll(name, "_", ""), norm) {
                found = true
                break
            }
        }
        if !found {
            return fmt.Errorf("no struct field for column %q", c)
        }
    }
    return nil
}

When it happens

Trigger: Calling databaseio.Read (or Query) with a struct type whose exported fields do not cover all columns returned by the SQL query/statement — e.g. the query selects a column the struct lacks, the column name differs beyond the snake_case/case-insensitive heuristics, or the matching field is unexported and therefore skipped.

Common situations: Schema drift: a teammate added a DB column (or a SELECT *) but the Go struct wasn't updated; DBAs use naming conventions like camelCase-with-hyphens or quoted mixed-case identifiers the matcher can't normalize; forgetting the `column:"db_name"` tag when Go naming and SQL naming diverge; using an unexported field thinking it would still map.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• bad decode
• bad encode
• bad return type for
• base map cannot be nil
• beam.RegisterSchemaProvider: unsupported type kind for…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/39430a79c3d5eea9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/databaseio/util.go:52

		fieldName := recordType.Field(i).Name
		indexedFields[fieldName] = i
		indexedFields[strings.ToLower(fieldName)] = i //to account for various matching strategies
		aTag := recordType.Field(i).Tag
		if column := aTag.Get("column"); column != "" {
			indexedFields[column] = i
		}
	}
	var mappedFieldIndex = make([]int, len(columns))
	for i, column := range columns {
		fieldIndex, ok := indexedFields[column]
		if !ok {
			fieldIndex, ok = indexedFields[strings.ToLower(column)]
		}
		if !ok {
			fieldIndex, ok = indexedFields[strings.Replace(strings.ToLower(column), "_", "", strings.Count(column, "_"))]
		}
		if !ok {
			return nil, errors.Errorf("failed to matched a %v field for SQL column: %v", recordType, column)
		}
		mappedFieldIndex[i] = fieldIndex
	}
	return mappedFieldIndex, nil
}

func asDereferenceSlice(aSlice []any) {
	for i, value := range aSlice {
		if value == nil {
			continue
		}
		aSlice[i] = reflect.ValueOf(value).Elem().Interface()

	}
}

func asMap(keys []string, values []any) map[string]any {
	var result = make(map[string]any)

View on GitHub (pinned to 12126d8942)