apache/beam · error

failed to query

Error message

failed to query: %v

What it means

Raised when query.Query() fails to execute the prepared `SELECT ... WHERE 1 = 0` schema-probe statement against the target table. The statement was prepared successfully, so the SQL is valid, but executing it failed — typically a connectivity, timeout, or runtime SQL error from the driver. The writer needs the (empty) result set only to read column metadata before doing INSERTs.

Solutions

1. Check database availability and network connectivity from the pipeline workers.
2. Retry the pipeline or re-run the failing bundle; transient connection drops are the most common cause.
3. Adjust DSN connection parameters (timeouts, keepalives) so idle prepared connections aren't dropped.
4. Inspect the wrapped driver error for the exact execution failure (e.g. 'connection refused', 'too many connections').

Example fix

// before
db, err := sql.Open(driver, dsn) // no timeouts; stale conns fail at Query
// after
db, err := sql.Open(driver, dsn)
db.SetConnMaxLifetime(5 * time.Minute) // recycle connections before they go stale

if err := db.PingContext(ctx); err != nil { return fmt.Errorf("database unreachable before write: %w", err) }

if err != nil {
    if isTransientNetErr(err) { // e.g. net.Error Timeout, driver 'bad connection'
        return err // let Beam retry the bundle
    }
    return errors.Wrapf(err, "permanent failure querying table %q", table)
}

When it happens

Trigger: Connection dropped or timed out between Prepare and Query; database became unavailable mid-pipeline; driver-level runtime error executing the probe query; the statement was prepared on a pool connection that later went stale.

Common situations: Long-running Beam pipelines where idle DB connections are reaped by a firewall/load balancer; transient network blips during bundle execution; database restarts; connection-pool exhaustion causing query execution failures.

Understand the failure class

Background: "query failed", "%w: SQL error" — wrapped database query errors in Go libraries explained — this error's family across 3 libraries.

Related errors

• Attempting to create database
• Attempting to create database
• Attempting to drop database
• Cannot use database: ' ' not found.
• Database operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4ebcaa4e68241d11. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/databaseio/database.go:175

	//TODO move DB Open and Close to Setup and Teardown methods or StartBundle and FinishBundle
	db, err := sql.Open(f.Driver, f.Dsn)
	if err != nil {
		return errors.Wrapf(err, "failed to open database: %v", f.Driver)
	}
	defer db.Close()
	projection := "*"
	if len(f.Columns) > 0 {
		projection = strings.Join(f.Columns, ",")
	}
	dql := fmt.Sprintf("SELECT %v FROM  %v WHERE 1 = 0", projection, f.Table)
	query, err := db.Prepare(dql)
	if err != nil {
		return errors.Wrapf(err, "failed to prepare query: %v", f.Table)
	}
	defer query.Close()
	rows, err := query.Query()
	if err != nil {
		return errors.Wrapf(err, "failed to query: %v", f.Table)
	}
	columns, err := rows.Columns()
	if err != nil {
		return errors.Wrapf(err, "failed to discover column: %v", f.Table)
	}
	//TODO move to Setup methods
	mapper, err := newWriterRowMapper(columns, f.Type.T)
	if err != nil {
		return errors.WithContext(err, "creating row mapper")
	}
	writer, err := newWriter(f.Driver, f.BatchSize, f.Table, columns)
	if err != nil {
		return err
	}
	var val beam.X
	for iter(&val) {
		var row []any
		var data map[string]any

View on GitHub (pinned to 12126d8942)