apache/beam · error · UnsupportedOperationException

Cannot read format

Error message

Cannot read format: {}

What it means

ScanTaskReader.advance() builds an iterator over a data file's records and only implements readers for the file formats it knows (AVRO, PARQUET via specific builders). If the task's DataFile has a format outside the handled switch cases, it throws UnsupportedOperationException with the file's format. It means the connector cannot read this type of Iceberg data file.

Solutions

1. Rewrite the table's files to a supported format: CALL catalog.system.rewrite_data_files(table => '...', options => map('format','parquet')).
2. Set the table's write format to parquet/avro (write.format.default) and re-ingest affected data.
3. If ORC support is required, upgrade the Beam IO iceberg connector or contribute/implement an ORC reader branch.

Example fix

// before
table.updateProperties().set("write.format.default", "orc").commit();
// after
table.updateProperties().set("write.format.default", "parquet").commit();

for (FileScanTask task : tasks) {
  FileFormat fmt = task.file().format();
  if (!fmt.equals(FileFormat.AVRO) && !fmt.equals(FileFormat.PARQUET)) {
    throw new IllegalStateException("Table contains unsupported file format: " + fmt + "; rewrite to parquet/avro first");
  }
}

try {
  reader.start();
} catch (UnsupportedOperationException e) {
  // trigger a rewrite job for the offending file format, then retry the pipeline
  throw new IllegalStateException("Unsupported data file format; rewrite table files", e);
}

When it happens

Trigger: A CombinedScanTask file whose FileFormat is not one of the implemented cases (e.g. ORC) is encountered while advancing the reader in ScanTaskReader.

Common situations: The Iceberg table contains ORC-written files but the pipeline expects AVRO/Parquet-only support; a table was populated by another engine using an unsupported format; format migration left mixed-format files.

Related errors

• BATCH scan not supported
• Encountered an error when creating a RecordWriter for table
• Encountered an error when parsing filter
• ORC file format not currently supported.
• smallBatchTag and largeBatchTag must be different

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5cba244818f92bee. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ScanTaskReader.java:181

          iterable = parquetReader.build();
          break;
        case AVRO:
          LOG.info("Preparing Avro input.");
          Avro.ReadBuilder avroReader =
              Avro.read(input)
                  .split(fileTask.start(), fileTask.length())
                  .project(requiredSchema)
                  .createReaderFunc(
                      fileSchema -> DataReader.create(requiredSchema, fileSchema, idToConstants));

          if (nameMapping != null) {
            avroReader.withNameMapping(NameMappingParser.fromJson(nameMapping));
          }

          iterable = avroReader.build();
          break;
        default:
          throw new UnsupportedOperationException("Cannot read format: " + file.format());
      }
      GenericDeleteFilter deleteFilter =
          new GenericDeleteFilter(
              checkStateNotNull(io), fileTask, fileTask.schema(), requiredSchema);
      iterable = deleteFilter.filter(iterable);

      iterable = ReadUtils.maybeApplyFilter(iterable, source.getScanConfig());
      currentIterator = iterable.iterator();
    } while (true);

    return false;
  }

  @Override
  public Row getCurrent() throws NoSuchElementException {
    if (current == null) {
      throw new NoSuchElementException();
    }

View on GitHub (pinned to 12126d8942)