{"record":{"id":"2142e1692a13dc40","repo":"apache/beam","slug":"missing-avro-file-signature-fileresource","errorCode":null,"errorMessage":"Missing Avro file signature: ${fileResource}","messagePattern":"Missing Avro file signature: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSource.java","lineNumber":491,"sourceCode":"  @VisibleForTesting\n  static AvroMetadata readMetadataFromFile(ResourceId fileResource) throws IOException {\n    String codec = null;\n    String schemaString = null;\n    byte[] syncMarker;\n    try (InputStream stream = Channels.newInputStream(FileSystems.open(fileResource))) {\n      BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(stream, null);\n\n      // The header of an object container file begins with a four-byte magic number, followed\n      // by the file metadata (including the schema and codec), encoded as a map. Finally, the\n      // header ends with the file's 16-byte sync marker.\n      // See https://avro.apache.org/docs/1.7.7/spec.html#Object+Container+Files for details on\n      // the encoding of container files.\n\n      // Read the magic number.\n      byte[] magic = new byte[DataFileConstants.MAGIC.length];\n      decoder.readFixed(magic);\n      if (!Arrays.equals(magic, DataFileConstants.MAGIC)) {\n        throw new IOException(\"Missing Avro file signature: \" + fileResource);\n      }\n\n      // Read the metadata to find the codec and schema.\n      ByteBuffer valueBuffer = ByteBuffer.allocate(512);\n      long numRecords = decoder.readMapStart();\n      while (numRecords > 0) {\n        for (long recordIndex = 0; recordIndex < numRecords; recordIndex++) {\n          String key = decoder.readString();\n          // readBytes() clears the buffer and returns a buffer where:\n          // - position is the start of the bytes read\n          // - limit is the end of the bytes read\n          valueBuffer = decoder.readBytes(valueBuffer);\n          byte[] bytes = new byte[valueBuffer.remaining()];\n          valueBuffer.get(bytes);\n          if (key.equals(DataFileConstants.CODEC)) {\n            codec = new String(bytes, StandardCharsets.UTF_8);\n          } else if (key.equals(DataFileConstants.SCHEMA)) {\n            schemaString = new String(bytes, StandardCharsets.UTF_8);","sourceCodeStart":473,"sourceCodeEnd":509,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSource.java#L473-L509","documentation":"AvroSource validates that the file it is reading begins with Avro's container-file magic bytes (Obj\\u0001). If the fixed-size magic read from the file does not match, it throws IOException — the file is not an Avro object container file.","triggerScenarios":"Reading a file matched by the source's filepattern that is not an Avro container file (plain Avro JSON, text, gzipped data, or a truncated/corrupt file).","commonSituations":"Glob patterns accidentally matching non-Avro files (e.g. .avsc schema files or _SUCCESS markers), partial uploads, or files written without the Avro container format.","solutions":["Narrow the filepattern so it only matches real Avro container files (e.g. *.avro, not *.avsc)","Inspect the first bytes of the offending file to confirm it starts with Obj\\u0001","Re-generate or re-upload corrupted/truncated files"],"exampleFix":"// before\nAvroIO.readGenericRecords(schema).from(\"output/*.avsc\");\n// after\nAvroIO.readGenericRecords(schema).from(\"output/part-*.avro\");","handlingStrategy":"try-catch","validationCode":"try (InputStream in = fs.open(path)) {\n  byte[] magic = new byte[4];\n  if (in.read(magic) != 4 || magic[0] != 'O' || magic[1] != 'b' || magic[2] != 'j' || magic[3] != 1)\n    throw new IOException(path + \" is not an Avro container file\");\n}","typeGuard":null,"tryCatchPattern":"try { source.readMetadataFromFile(resource); } catch (IOException e) { LOG.warn(\"Skipping non-Avro file: {}\", resource); }","preventionTips":["Use filepatterns that cannot match schema files or marker files","Verify file integrity after uploads before running the pipeline","Check magic bytes when ingesting third-party data"],"tags":["avro","file-format","corruption"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}