What causes "failed to scan %v" in apache/beam?

List columns explicitly in the SELECT so column order/count always matches the struct fields.

failed to scan

Error message

failed to scan %v

What it means

For each row, the DoFn calls rows.Scan(rowValues...) into destination pointers produced by the row mapper. If a column's value cannot be converted into the mapped destination type (NULL into non-nullable field, string into int, wrong column count), it fails with "failed to scan %v" wrapping the query text. This is a schema/type mismatch between the SQL result set and the Go element type.

Solutions

List columns explicitly in the SELECT so column order/count always matches the struct fields.
Change struct fields for nullable columns to pointers or sql.NullString/sql.NullInt64 etc.
Align field types with column types (e.g. scan numeric columns into appropriate int64/float64, decode JSON/text columns manually).
Use a MapLoader/SliceLoader implementation to handle dynamic or wide result sets instead of a fixed struct.

Example fix

// before
type user struct {
  Email string // column is NULLable
}

// after
type user struct {
  Email sql.NullString // or *string
}

Defensive patterns

Strategy: type-guard

Validate before calling

cols, _ := rows.Columns()
if len(cols) != expectedFieldCount { return fmt.Errorf("column count %d != struct fields %d", len(cols), expectedFieldCount) }

Type guard

func isNullableCompatible(field reflect.StructField) bool {
	t := field.Type
	return t.Kind() == reflect.Ptr || strings.HasPrefix(t.Name(), "Null") // *T or sql.Null*
}

Try / catch

if err := rows.Scan(dest...); err != nil {
	return fmt.Errorf("failed to scan %q: %w", query, err) // log columns+row for diagnosis
}

Prevention

Avoid SELECT *; list columns explicitly to keep order/count stable.
Map nullable columns to pointers or sql.Null* types.
Add a test that scans a representative row set into the element struct.

When it happens

Trigger: rows.Scan returns an error: destination slice length differs from the column count, NULL scanned into a non-pointer/non-sql.Nullable field, incompatible column value types, or unsupported column types (e.g. arrays, JSON) for the chosen mapper.

Common situations: Struct fields not matching SELECT column order/count after editing the query; nullable columns mapped to value types instead of pointers or sql.Null*; driver returning types (e.g. []byte for text) that don't fit the field; SELECT * picking up new columns after migrations.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5152c9320b62d88c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/databaseio/database.go:109

		reflectRow := reflect.New(f.Type.T)
		row := reflectRow.Interface() // row : *T
		if mapper == nil {
			columns, err = rows.Columns()
			if err != nil {
				return err
			}
			columnsTypes, _ := rows.ColumnTypes()
			if mapper, err = newQueryMapper(columns, columnsTypes, f.Type.T); err != nil {
				return errors.WithContext(err, "creating rowValues mapper")
			}
		}
		rowValues, err := mapper(reflectRow)
		if err != nil {
			return err
		}
		err = rows.Scan(rowValues...)
		if err != nil {
			return errors.Wrapf(err, "failed to scan %v", f.Query)
		}
		if loader, ok := row.(MapLoader); ok {
			asDereferenceSlice(rowValues)
			loader.LoadMap(asMap(columns, rowValues))
		} else if loader, ok := row.(SliceLoader); ok {
			asDereferenceSlice(rowValues)
			loader.LoadSlice(rowValues)
		}
		emit(reflect.ValueOf(row).Elem().Interface()) // emit(*row)
	}
	return nil
}

// Write writes the elements of the given PCollection<T> to database, if columns left empty all table columns are used to insert into, otherwise selected
func Write(s beam.Scope, driver, dsn, table string, columns []string, col beam.PCollection) {
	t := col.Type().Type()
	s = s.Scope(driver + ".Write")
	pre := beam.AddFixedKey(s, col)

View on GitHub (pinned to 12126d8942)