apache/beam · error · IllegalArgumentException

Unable to find value #${index}

Error message

Unable to find value #${index}

What it means

While generating code to read a field from a Beam Row inside the Calc operator, getBeamField validates the column index against the schema's field count. An index outside [0, fieldCount) means the generated code and the row schema disagree, so IllegalArgumentException is thrown. This is an internal consistency check between Calcite's row type and the Beam input schema.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java:536

      this.inputSchema = inputSchema;
      this.referencedColumns = new TreeSet<>();
    }

    FieldAccessDescriptor getFieldAccess() {
      return FieldAccessDescriptor.withFieldIds(this.referencedColumns);
    }

    @Override
    public Expression field(BlockBuilder list, int index, Type storageType) {
      this.referencedColumns.add(index);
      return getBeamField(list, index, input, inputSchema, true);
    }

    // Read field from Beam Row
    private static Expression getBeamField(
        BlockBuilder list, int index, Expression input, Schema schema, boolean useByteString) {
      if (index >= schema.getFieldCount() || index < 0) {
        throw new IllegalArgumentException("Unable to find value #" + index);
      }

      final Expression expression = list.append(list.newName("current"), input);
      final Field field = schema.getField(index);
      final FieldType fieldType = field.getType();
      final Expression fieldName = Expressions.constant(field.getName());
      Expression value = getBeamField(list, expression, fieldName, fieldType);

      return toCalciteValue(value, fieldType, useByteString);
    }

    private static Expression getBeamField(
        BlockBuilder list, Expression expression, Expression fieldName, FieldType fieldType) {
      final Expression value;
      switch (fieldType.getTypeName()) {
        case BYTE:
          return Expressions.call(expression, "getByte", fieldName);
        case INT16:

View on GitHub (pinned to 12126d8942)

Solutions

  1. Verify the PCollection registered for the table has a schema matching what the query selects (names/order/count)
  2. Check for schema drift between pipeline stages (e.g. upstream transform dropped a field)
  3. Simplify the query to isolate which column index is out of range (e.g. `SELECT *` first)
  4. Upgrade Beam if it looks like a codegen bug with a specific query shape

Example fix

// before
p.apply("t", rowsWithDroppedField).apply(SqlTransform.query("SELECT a, b, c FROM t"));
// after
// ensure rowsWithDroppedField schema contains fields a, b, c, or select only existing fields
p.apply(SqlTransform.query("SELECT a, b FROM t"));
Defensive patterns

Strategy: validation

Validate before calling

boolean schemaMatchesQuery(Schema s, int maxReferencedIndex) {
  return s.getFieldCount() > maxReferencedIndex;
}

Type guard

boolean columnIndexInBounds(Schema s, int i) {
  return i >= 0 && i < s.getFieldCount();
}

Try / catch

try {
  rows.apply(SqlTransform.query(sql));
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Unable to find value #")) {
    // schema/query mismatch: re-check registered table schema
  }
}

Prevention

When it happens

Trigger: Running a SQL query where the Calcite plan references a column ordinal that the incoming Beam Row schema doesn't have — typically after a mismatched input PCollection schema, aliased/renamed columns in a subquery, or a codegen bug.

Common situations: Passing a PCollection with fewer fields than the query expects (wrong table registered to the same name); schema evolved upstream (column dropped) while the query still references it; UNION/subquery column-count mismatches.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b46f4683eb18ba41. Report an issue: GitHub.