apache/beam · error · IOException

Missing Avro file signature

Error message

Missing Avro file signature: ${fileResource}

What it means

AvroSource validates that the file it is reading begins with Avro's container-file magic bytes (Obj\u0001). If the fixed-size magic read from the file does not match, it throws IOException — the file is not an Avro object container file.

Solutions

1. Narrow the filepattern so it only matches real Avro container files (e.g. *.avro, not *.avsc)
2. Inspect the first bytes of the offending file to confirm it starts with Obj\u0001
3. Re-generate or re-upload corrupted/truncated files

Example fix

// before
AvroIO.readGenericRecords(schema).from("output/*.avsc");
// after
AvroIO.readGenericRecords(schema).from("output/part-*.avro");

try (InputStream in = fs.open(path)) {
  byte[] magic = new byte[4];
  if (in.read(magic) != 4 || magic[0] != 'O' || magic[1] != 'b' || magic[2] != 'j' || magic[3] != 1)
    throw new IOException(path + " is not an Avro container file");
}

try { source.readMetadataFromFile(resource); } catch (IOException e) { LOG.warn("Skipping non-Avro file: {}", resource); }

When it happens

Trigger: Reading a file matched by the source's filepattern that is not an Avro container file (plain Avro JSON, text, gzipped data, or a truncated/corrupt file).

Common situations: Glob patterns accidentally matching non-Avro files (e.g. .avsc schema files or _SUCCESS markers), partial uploads, or files written without the Avro container format.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• A schema must be provided when writing to BigQuery using…
• An explicit schema is required to write non-schema'd…
• Avro only supports maps with string keys
• AVRO schema doesn't match row schema. Row schema
• AvroCoder for GenericRecord requires a schema

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2142e1692a13dc40. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSource.java:491

  @VisibleForTesting
  static AvroMetadata readMetadataFromFile(ResourceId fileResource) throws IOException {
    String codec = null;
    String schemaString = null;
    byte[] syncMarker;
    try (InputStream stream = Channels.newInputStream(FileSystems.open(fileResource))) {
      BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(stream, null);

      // The header of an object container file begins with a four-byte magic number, followed
      // by the file metadata (including the schema and codec), encoded as a map. Finally, the
      // header ends with the file's 16-byte sync marker.
      // See https://avro.apache.org/docs/1.7.7/spec.html#Object+Container+Files for details on
      // the encoding of container files.

      // Read the magic number.
      byte[] magic = new byte[DataFileConstants.MAGIC.length];
      decoder.readFixed(magic);
      if (!Arrays.equals(magic, DataFileConstants.MAGIC)) {
        throw new IOException("Missing Avro file signature: " + fileResource);
      }

      // Read the metadata to find the codec and schema.
      ByteBuffer valueBuffer = ByteBuffer.allocate(512);
      long numRecords = decoder.readMapStart();
      while (numRecords > 0) {
        for (long recordIndex = 0; recordIndex < numRecords; recordIndex++) {
          String key = decoder.readString();
          // readBytes() clears the buffer and returns a buffer where:
          // - position is the start of the bytes read
          // - limit is the end of the bytes read
          valueBuffer = decoder.readBytes(valueBuffer);
          byte[] bytes = new byte[valueBuffer.remaining()];
          valueBuffer.get(bytes);
          if (key.equals(DataFileConstants.CODEC)) {
            codec = new String(bytes, StandardCharsets.UTF_8);
          } else if (key.equals(DataFileConstants.SCHEMA)) {
            schemaString = new String(bytes, StandardCharsets.UTF_8);

View on GitHub (pinned to 12126d8942)