apache/beam · error · IllegalArgumentException

ArrowFlightIO.write(): e.getMessage()

Error message

ArrowFlightIO.write(): e.getMessage()

What it means

ArrowFlightIO.validateWriteSchema converts the Beam Schema to an Arrow schema; if that conversion throws IllegalArgumentException (a Beam type unsupported by Arrow conversion), it is rethrown as 'ArrowFlightIO.write(): <reason>'. The sink's schema cannot be represented in Arrow.

Solutions

1. Inspect the chained cause to find the unsupported field type and remove or map it to a supported type.
2. Convert unsupported fields to strings or primitives before writing.
3. Upgrade Beam to a version whose Arrow schema translation supports your types.
4. Test schema translation locally with ArrowConversion.ArrowSchemaTranslator.toArrowSchema(schema) before submitting the pipeline.

Example fix

// before
Row with LogicalType customField...
.apply(ArrowFlightIO.write());
// after
row.set("customField", customField.toString()); // map to supported STRING type
.apply(ArrowFlightIO.write());

// before submitting, verify the Beam schema translates to Arrow
ArrowConversion.ArrowSchemaTranslator.toArrowSchema(pcoll.getSchema());

try {
  pipeline.apply(ArrowFlightIO.write());
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("ArrowFlightIO.write():")) {
    log.error("Schema contains a Beam type not representable in Arrow", e);
  }
}

When it happens

Trigger: PCollection<Row> with a schema containing types that ArrowConversion.ArrowSchemaTranslator.toArrowSchema cannot map, passed to ArrowFlightIO.write().

Common situations: Nested/logical or custom logical types in the Beam schema, schema produced dynamically from data that includes exotic field types, Beam/Arrow version mismatches.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Arrow schema conversion does not support Beam type
• BeamRowMapper does not have support for fields of type
• can't generate row coder for type
• Cannot find Spanner table.
• cannot make schema for type

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/241ff6d187a0d90d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/arrow-flight/src/main/java/org/apache/beam/sdk/io/arrowflight/ArrowFlightIO.java:143

  private static byte[] copyToken(byte[] token) {
    return Arrays.copyOf(checkNotNull(token, "token"), token.length);
  }

  private static CallOption[] callOptions(byte @Nullable [] token) {
    if (token == null) {
      return new CallOption[0];
    }
    FlightCallHeaders headers = new FlightCallHeaders();
    headers.insert("authorization", "Bearer " + new String(token, StandardCharsets.UTF_8));
    return new CallOption[] {new HeaderCallOption(headers)};
  }

  private static void validateWriteSchema(Schema schema) {
    try {
      ArrowConversion.ArrowSchemaTranslator.toArrowSchema(schema);
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException("ArrowFlightIO.write(): " + e.getMessage(), e);
    }
  }

  public static Read read() {
    return new AutoValue_ArrowFlightIO_Read.Builder().setPort(47470).setUseTls(false).build();
  }

  public static Write write() {
    return new AutoValue_ArrowFlightIO_Write.Builder()
        .setPort(47470)
        .setUseTls(false)
        .setBatchSize(1024)
        .build();
  }

  /**
   * Creates a {@link FlightClient} from the given connection parameters.
   *

View on GitHub (pinned to 12126d8942)