apache/beam · error · RuntimeException

Error while determining columns from table: ${getTable()}

Error message

Error while determining columns from table: ${getTable()}

What it means

JdbcIO.withSchema() reads the table's metadata at pipeline setup time to derive a Beam Schema. It executes `SELECT * FROM <table>` via connection.prepareStatement() and converts the ResultSetMetaData to a Beam Schema with SchemaUtil.toBeamSchema(). If any SQLException occurs during that round trip (prepare or metadata fetch), it is wrapped in this RuntimeException.

Source

Thrown at sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java:2526

          JdbcUtil.generateStatement(
              checkStateNotNull(getTable()),
              fields.stream().map(FieldWithIndex::getField).collect(Collectors.toList())));
    }

    // Spotbugs seems to not understand the multi-statement try-with-resources
    @SuppressFBWarnings("OBL_UNSATISFIED_OBLIGATION")
    private List<SchemaUtil.FieldWithIndex> getFilteredFields(Schema schema) {
      Schema tableSchema;

      try (Connection connection =
              checkStateNotNull(getDataSourceProviderFn()).apply(null).getConnection();
          PreparedStatement statement =
              connection.prepareStatement(String.format("SELECT * FROM %s", getTable()))) {
        ResultSetMetaData metadata =
            checkStateNotNull(statement.getMetaData(), "could not get statement metadata");
        tableSchema = SchemaUtil.toBeamSchema(metadata);
      } catch (SQLException e) {
        throw new RuntimeException("Error while determining columns from table: " + getTable(), e);
      }

      checkState(
          tableSchema.getFieldCount() >= schema.getFieldCount(),
          String.format(
              "Input schema has more fields (%s) than actual table (%s).%n\t"
                  + "Input schema fields: %s | Table fields: %s",
              tableSchema.getFieldCount(),
              schema.getFieldCount(),
              schema.getFields().stream()
                  .map(Schema.Field::getName)
                  .collect(Collectors.joining(", ")),
              tableSchema.getFields().stream()
                  .map(Schema.Field::getName)
                  .collect(Collectors.joining(", "))));

      // filter out missing fields from output table
      List<Schema.Field> missingFields =

View on GitHub (pinned to 12126d8942)

Solutions

Verify the table name is correct and fully qualified (database/schema.table) for the target connection.
Confirm the connection config (URL, user, password, driver jars) by connecting manually with the same credentials.
Grant the configured DB user SELECT and metadata-reading privileges on the table.
Catch the RuntimeException and inspect the wrapped SQLException cause for the underlying reason (connection refused vs. table not found vs. permission denied).

Example fix

// before
JdbcIO.<Row>readWithPartitions().withTable("users") // table lives in another schema
// after
JdbcIO.<Row>readWithPartitions().withTable("mydb.public.users")

Defensive patterns

Strategy: try-catch

Validate before calling

// before building the pipeline
if (table == null || !table.contains(".")) {
  throw new IllegalArgumentException("Use a fully qualified table name: db.schema.table");
}
try (Connection c = DriverManager.getConnection(url, user, pass);
     PreparedStatement ps = c.prepareStatement("SELECT * FROM " + table)) {
  ps.getMetaData(); // probes existence + permissions
}

Try / catch

try {
  pipeline.apply(JdbcIO.<Row>readWithPartitions()...);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Error while determining columns from table")) {
    Throwable cause = e.getCause(); // inspect SQLException for table-not-found vs access denied
  }
  throw e;
}

Prevention

Always qualify table names with database/schema.
Smoke-test connectivity and SELECT permissions from the same network as the runner before launching jobs.
Package the correct JDBC driver jar via withDriverJars() or classpath.

When it happens

Trigger: Calling JdbcIO.<Row>readWithPartitions/withSchema where the configured table name is wrong or unqualified, the table does not exist, the configured user lacks SELECT privileges on the table, or the connection is misconfigured/unreachable while JdbcIO runs `SELECT * FROM <table>` to infer the schema.

Common situations: Typo in table name or missing schema/database qualifier (e.g. need `mydb.public.users`), running in a VPC/network where the DB is not reachable from the worker, credentials without metadata permissions, or driver not on classpath so connection setup fails.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/31219a6114b3319b. Report an issue: GitHub.