What causes "failed to discover column: %v" in apache/beam?

Check the wrapped driver error and verify the driver fully implements the database/sql Rows API.

failed to discover column

Error message

failed to discover column: %v

What it means

Raised when rows.Columns() fails on the (empty) result set of the schema-probe query. The library reads the result-set column names to build a row mapper for inserts; this failure means the driver could not report the result-set metadata. It is rare compared to prepare/query failures and usually indicates a driver bug or a broken connection when fetching metadata.

Solutions

Check the wrapped driver error and verify the driver fully implements the database/sql Rows API.
Verify network stability between the worker and the database; retry the failing bundle.
Test rows.Columns() against the same driver in a small standalone program to isolate a driver bug.
Update the driver to a version with fixed metadata handling.

Example fix

// before
rows, _ := db.Query("SELECT * FROM users WHERE 1=0")
// metadata fetch may fail on broken conns
// after
if err := db.PingContext(ctx); err != nil { /* reconnect / fail fast */ }
rows, err := db.Query("SELECT * FROM users WHERE 1=0")

Defensive patterns

Strategy: try-catch

Validate before calling

if err := db.PingContext(ctx); err != nil { return err }
rows, err := db.Query("SELECT * FROM " + table + " WHERE 1 = 0")
if err != nil { return err }
if _, err := rows.Columns(); err != nil { return fmt.Errorf("driver cannot report metadata: %w", err) }

Try / catch

if err != nil {
    return errors.Wrapf(err, "could not read column metadata for table %q; suspect driver bug or dropped connection", table)
}

Prevention

Use a mature, well-maintained database/sql driver for your backend.
Smoke-test rows.Columns() against your driver in a small program before running the pipeline.
Keep connection lifetimes bounded so metadata calls don't hit dead connections.
Pin a known-good driver version; upgrade deliberately after testing.

When it happens

Trigger: Connection lost between query execution and metadata retrieval; a database/sql driver whose Rows implementation returns an error from Columns(); fetching metadata on an already-closed or exhausted connection.

Common situations: Buggy or third-party drivers with incomplete database/sql implementations; network interruptions between executing the probe query and reading its metadata; extremely short connection lifetimes set via SetConnMaxLifetime.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ff40bcbd34f99170. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/databaseio/database.go:179

	}
	defer db.Close()
	projection := "*"
	if len(f.Columns) > 0 {
		projection = strings.Join(f.Columns, ",")
	}
	dql := fmt.Sprintf("SELECT %v FROM  %v WHERE 1 = 0", projection, f.Table)
	query, err := db.Prepare(dql)
	if err != nil {
		return errors.Wrapf(err, "failed to prepare query: %v", f.Table)
	}
	defer query.Close()
	rows, err := query.Query()
	if err != nil {
		return errors.Wrapf(err, "failed to query: %v", f.Table)
	}
	columns, err := rows.Columns()
	if err != nil {
		return errors.Wrapf(err, "failed to discover column: %v", f.Table)
	}
	//TODO move to Setup methods
	mapper, err := newWriterRowMapper(columns, f.Type.T)
	if err != nil {
		return errors.WithContext(err, "creating row mapper")
	}
	writer, err := newWriter(f.Driver, f.BatchSize, f.Table, columns)
	if err != nil {
		return err
	}
	var val beam.X
	for iter(&val) {
		var row []any
		var data map[string]any
		if writer, ok := val.(Writer); ok {
			if data, err = writer.SaveData(); err == nil {
				row = make([]any, len(columns))
				for i, column := range columns {

View on GitHub (pinned to 12126d8942)