apache/beam · error · ParquetDecodingException

Can not read value at

Error message

Can not read value at %d in block %d in file %s

What it means

ParquetIO.ReadFiles wraps any RuntimeException thrown while decoding a Parquet record into a ParquetDecodingException with the row, block, and file coordinates. It means the file contains a record that could not be parsed with the configured Avro schema/function, not that the file itself is unreadable.

Solutions

1. Verify the Avro schema passed to ParquetIO.readFiles matches the writer schema of the file (use the file's own schema via AvroParquetReader.getSchema).
2. Inspect the file at the reported row/block with parquet-tools / parquet cat to find the malformed record.
3. Wrap your parseFn logic defensively so problematic records are logged/skipped instead of thrown.
4. Re-export or regenerate the source parquet file if it is corrupt.

Example fix

// before
ParquetIO.readFiles(schema).withParseFn(parseFn);
// after
ParquetIO.readFiles(file.getSchema()).withParseFn(safeParseFn); // safeParseFn returns null/logs on bad records

// Pre-validate file vs schema
Schema parserSchema = schema; // Avro schema passed to readFiles
Schema fileSchema = AvroParquetReader.builder(parserSchema).build(new Path(file)).getSchema();
if (!fileSchema.equals(parserSchema)) { log.warn("schema mismatch for " + file); }

// ParquetIO throws ParquetDecodingException wrapping the cause
try {
  pipeline.apply(ParquetIO.readFiles(schema));
} catch (ParquetDecodingException e) {
  log.error("decode failed at row {} block {} file {}", e); // inspect e.getCause()
}

When it happens

Trigger: Calling ParquetIO.readFiles(schema) and the pipeline hits a row whose decode via parseFn (e.g. GenericRecord-to-Avro conversion) throws a RuntimeException.

Common situations: Schema mismatch between the read schema and the file's actual schema; corrupted parquet files; custom parse functions throwing on unexpected field values (nulls, incompatible unions).

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• A schema is required to write non-schema'd data.
• array len mismatch. decoding
• Couldn't set the specified Avro data model
• decodeMultiChunkStream chunk size decoding failed
• decodeMultiChunkStream invalid chunk size

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a4a8037f9894df5c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java:827

                  LOG.debug(
                      "record is filtered out by reader in block {} in file {}",
                      currentBlock,
                      file.toString());
                  continue;
                }
                if (recordReader.shouldSkipCurrentRecord()) {
                  // this record is being filtered via the filter2 package
                  LOG.debug(
                      "skipping record at {} in block {} in file {}",
                      currentRow,
                      currentBlock,
                      file.toString());
                  continue;
                }
                outputReceiver.output(parseFn.apply(record));
              } catch (RuntimeException e) {

                throw new ParquetDecodingException(
                    format(
                        "Can not read value at %d in block %d in file %s",
                        currentRow, currentBlock, file.toString()),
                    e);
              }
            }
            LOG.debug(
                "Finish processing {} rows from block {} in file {}",
                currentRow,
                currentBlock - 1,
                file.toString());
          }
        }
      }

      public Configuration getConfWithModelClass() throws ReflectiveOperationException {
        Configuration conf = SerializableConfiguration.newConfiguration(configuration);
        GenericData model = buildModelObject(modelClass);

View on GitHub (pinned to 12126d8942)