apache/beam · error · RuntimeException

Unable run pipeline with CREATE IF NEEDED disposition.

Error message

Unable run pipeline with CREATE IF NEEDED disposition.

What it means

checkResultIfTableExists wraps a SQLException thrown while reading the table-existence result set for the CREATE IF NEEDED disposition. The existence probe failed at the JDBC level, so the pipeline cannot determine whether to create the table.

Source

Thrown at sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/SnowflakeBatchServiceImpl.java:262

          if (!checkResultIfTableExists(resultSet)) {
            try {
              createTable(dataSource, table, tableSchema);
            } catch (SQLException e) {
              throw new RuntimeException("Unable to create table.", e);
            }
          }
        });
  }

  private static boolean checkResultIfTableExists(ResultSet resultSet) {
    try {
      if (resultSet.next()) {
        return checkIfResultIsTrue(resultSet);
      } else {
        throw new RuntimeException("Unable run pipeline with CREATE IF NEEDED - no response.");
      }
    } catch (SQLException e) {
      throw new RuntimeException("Unable run pipeline with CREATE IF NEEDED disposition.", e);
    }
  }

  private void createTable(DataSource dataSource, String table, SnowflakeTableSchema tableSchema)
      throws SQLException {
    checkArgument(
        tableSchema != null,
        "The CREATE_IF_NEEDED disposition requires schema if table doesn't exists");
    String query = String.format("CREATE TABLE %s (%s);", table, tableSchema.sql());
    runConnectionWithStatement(dataSource, query, null);
  }

  private static boolean checkIfResultIsTrue(ResultSet resultSet) throws SQLException {
    int columnId = 1;
    return resultSet.getBoolean(columnId);
  }

  private static void runConnectionWithStatement(

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the wrapped SQLException cause for the root JDBC error.
2. Ensure any custom StatementExecutionSupplier does not close or consume the ResultSet before returning it.
3. Retry the pipeline; transient Snowflake errors typically clear on retry.
4. Validate credentials/warehouse state (suspended warehouse, expired session).

Example fix

// before: supplier consumes the ResultSet
rs.next(); // read it for logging, then return rs
// after: return it untouched
return statement.executeQuery("SHOW OBJECTS IN " + tableRef);

// Pre-flight: run the existence probe manually
// SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='...' AND table_name='...';

try { pipeline.run(); }
catch (RuntimeException e) {
  if (e.getCause() instanceof java.sql.SQLException sqlEx && isTransient(sqlEx)) retry();
  else throw e;
}

When it happens

Trigger: resultSet.next() or checkIfResultIsTrue throws SQLException during createTableIfNotExists — connection errors, closed result set, invalid cursor state while running the CREATE_IF_NEEDED preflight.

Common situations: Transient Snowflake connectivity issues; custom StatementExecutionSupplier closing the result set before Beam reads it; session/warehouse timeouts.

Understand the failure class

Background: "query failed", "%w: SQL error" — wrapped database query errors in Go libraries explained — this error's family across 3 libraries.

Related errors

• Unable run pipeline with EMPTY disposition.
• Beam JDBC connection has not been initialized
• Unable to parse statement
• Failed to infer Beam schema
• Unable to create prepared statement for type: ${type}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fdbde790a66f4e0a. Report an issue: GitHub.