apache/beam · error · java.lang.Exception

Failed to get number of partitions in the database

Error message

Failed to get number of partitions in the database

What it means

SingleStoreIO's partitioned read resolves the number of store partitions by querying information_schema.DISTRIBUTED_DATABASES for the target database. If the query returns no rows, the code cannot determine partition count and throws "Failed to get number of partitions in the database", aborting split planning.

Source

Thrown at sdks/java/io/singlestore/src/main/java/org/apache/beam/sdk/io/singlestore/SingleStoreIO.java:765

        for (long i = range.getFrom(); i < range.getTo(); i++) {
          receiver.output(new OffsetRange(i, i + 1));
        }
      }

      private int getNumPartitions() throws Exception {
        DataSource dataSource = dataSourceConfiguration.getDataSource();
        Connection conn = dataSource.getConnection();
        try {
          Statement stmt = conn.createStatement();
          try {
            ResultSet res =
                stmt.executeQuery(
                    String.format(
                        "SELECT num_partitions FROM information_schema.DISTRIBUTED_DATABASES WHERE database_name = %s",
                        SingleStoreUtil.escapeString(database)));
            try {
              if (!res.next()) {
                throw new Exception("Failed to get number of partitions in the database");
              }

              return res.getInt(1);
            } finally {
              res.close();
            }
          } finally {
            stmt.close();
          }
        } finally {
          conn.close();
        }
      }
    }

    @Override
    public void populateDisplayData(DisplayData.Builder builder) {
      super.populateDisplayData(builder);

View on GitHub (pinned to 12126d8942)

Solutions

Verify the database name in the read configuration matches an existing distributed database
Run SELECT num_partitions FROM information_schema.DISTRIBUTED_DATABASES manually with the same credentials to confirm a row exists
Check the JDBC URL points at the intended SingleStore cluster
If the DB is genuinely not distributed, use a non-partitioned read instead of withPartitions()

Example fix

// before
SingleStoreIO.<Row>read().withDataSourceConfiguration(cfg)
    .withQuery("SELECT * FROM mydb.t")
    .withPartitions() // database 'mydb' misnamed -> throws
// after
SingleStoreIO.<Row>read().withDataSourceConfiguration(cfg)
    .withQuery("SELECT * FROM sales.t") // 'sales' exists in DISTRIBUTED_DATABASES
    .withPartitions()

Defensive patterns

Strategy: validation

Validate before calling

try (Connection c = DriverManager.getConnection(url, user, pass); Statement s = c.createStatement(); ResultSet r = s.executeQuery("SELECT num_partitions FROM information_schema.DISTRIBUTED_DATABASES WHERE database_name = '" + db + "'")) { if (!r.next()) throw new IllegalStateException("DB not distributed or missing: " + db); }

Try / catch

try { pipeline.apply(SingleStoreIO.read().withPartitions()...); } catch (Exception e) { if (e.getMessage().contains("Failed to get number of partitions")) { /* fall back to non-partitioned read */ } throw e; }

Prevention

Confirm the database appears in information_schema.DISTRIBUTED_DATABASES before using withPartitions()
Validate config database names against the live cluster, not hardcoded values
Grant the connector user SELECT on information_schema

When it happens

Trigger: SingleStoreIO.read().withPartitions() (partition discovery) against a database that has no row in information_schema.DISTRIBUTED_DATABASES — typically when the configured database name doesn't exist, is misspelled, or points to a single-node/non-distributed database.

Common situations: Typos in the database name in SingleStoreIO.Read config; connecting to a single-store (non-distributed) deployment; running against a different cluster than intended via connection URL; insufficient privileges make the view appear empty.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/47464bb122ca6f40. Report an issue: GitHub.