What causes "Format of provided table schema is invalid" in apache/beam?

Pass valid JSON matching SnowflakeTableSchema, e.g. '[{"name":"id","type":"NUMBER"}]', with double quotes.

apache/beam · error · RuntimeException

Format of provided table schema is invalid

Error message

Format of provided table schema is invalid

What it means

WriteBuilder.setTableSchema() (Snowflake cross-language Java API) deserializes the tableSchema string into SnowflakeTableSchema with Jackson. If the string is not valid JSON or does not match the expected schema shape, the IOException is converted to a RuntimeException with this message, aborting cross-language pipeline expansion.

Solutions

Pass valid JSON matching SnowflakeTableSchema, e.g. '[{"name":"id","type":"NUMBER"}]', with double quotes.
On the Python side, use json.dumps(schema_list) before handing the value to the Java transform.
Validate the string with json.loads (Python) or a JSON linter before calling setTableSchema; check for shell/option escaping issues.

Example fix

// before
builder.setTableSchema("[{name: 'id', type: 'NUMBER'}]")
// after
builder.setTableSchema("[{\"name\":\"id\",\"type\":\"NUMBER\"}]")

Defensive patterns

Strategy: try-catch

Validate before calling

import json
columns = [{"name": "id", "type": "NUMBER"}]
json.dumps(columns)  # validate before passing to setTableSchema

Type guard

function isValidTableSchemaJson(s) {
  try {
    const v = JSON.parse(s);
    return Array.isArray(v) && v.every(c => typeof c.name === 'string' && typeof c.type === 'string');
  } catch { return false; }
}

Try / catch

try {
  builder.setTableSchema(tableSchemaJson);
} catch (RuntimeException e) {
  if (e.getMessage().contains("Format of provided table schema is invalid")) {
    // re-parse/repair the JSON on the producing side before retrying
  }
  throw e;
}

Prevention

Generate the schema string with json.dumps / a JSON serializer, never string concatenation or Python repr().
Validate the JSON with a parser on the producing side before handing it to the cross-language transform.
Beware double-escaping when passing JSON through pipeline options or shell commands.

When it happens

Trigger: Calling setTableSchema() with a non-JSON string, JSON that isn't an array of column objects, or malformed input from a Python/YAML side that wasn't converted with to_json.

Common situations: Python Beam users passing a Python list/dict repr (single quotes, unquoted keys) instead of JSON; hand-written schema strings with trailing commas or comments; double-escaping issues when the JSON travels through pipeline options.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/532f2e2b0e5ad085. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/crosslanguage/WriteBuilder.java:51

  "nullness" // TODO(https://github.com/apache/beam/issues/20497)
})
public class WriteBuilder
    implements ExternalTransformBuilder<
        WriteBuilder.Configuration, PCollection<List<byte[]>>, PDone> {

  /** Parameters class to expose the transform to an external SDK. */
  public static class Configuration extends CrossLanguageConfiguration {
    private SnowflakeTableSchema tableSchema;
    private CreateDisposition createDisposition;
    private WriteDisposition writeDisposition;

    public void setTableSchema(String tableSchema) {
      ObjectMapper mapper = new ObjectMapper();

      try {
        this.tableSchema = mapper.readValue(tableSchema, SnowflakeTableSchema.class);
      } catch (IOException e) {
        throw new RuntimeException("Format of provided table schema is invalid");
      }
    }

    public void setCreateDisposition(String createDisposition) {
      this.createDisposition = CreateDisposition.valueOf(createDisposition);
    }

    public void setWriteDisposition(String writeDisposition) {
      this.writeDisposition = WriteDisposition.valueOf(writeDisposition);
    }

    public SnowflakeTableSchema getTableSchema() {
      return tableSchema;
    }

    public CreateDisposition getCreateDisposition() {
      return createDisposition;
    }

View on GitHub (pinned to 12126d8942)