{"record":{"id":"3cc064cb26ee4583","repo":"apache/beam","slug":"failed-to-deserialize-missingpartitions","errorCode":null,"errorMessage":"Failed to deserialize missingPartitions: {}","messagePattern":"Failed to deserialize missingPartitions: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableDao.java","lineNumber":788,"sourceCode":"    Row row = dataClient.readRow(tableId, getFullDetectNewPartition(), missingPartitionsFilter);\n\n    if (row == null\n        || row.getCells(\n                MetadataTableAdminDao.CF_MISSING_PARTITIONS,\n                MetadataTableAdminDao.QUALIFIER_DEFAULT)\n            .isEmpty()) {\n      return missingPartitions;\n    }\n    ByteString serializedMissingPartition =\n        row.getCells(\n                MetadataTableAdminDao.CF_MISSING_PARTITIONS,\n                MetadataTableAdminDao.QUALIFIER_DEFAULT)\n            .get(0)\n            .getValue();\n    try {\n      missingPartitions = SerializationUtils.deserialize(serializedMissingPartition.toByteArray());\n    } catch (SerializationException | NullPointerException exception) {\n      LOG.warn(\"Failed to deserialize missingPartitions: {}\", exception.toString());\n    }\n    return missingPartitions;\n  }\n\n  /**\n   * Write to metadata table serialized missing partitions and how long they have been missing.\n   *\n   * @param missingPartitionDurations missing partitions and duration.\n   */\n  public void writeDetectNewPartitionMissingPartitions(\n      HashMap<ByteStringRange, Instant> missingPartitionDurations) {\n    long nowMicros = Instant.now().getMillis() * 1000L;\n    byte[] serializedMissingPartition = SerializationUtils.serialize(missingPartitionDurations);\n    RowMutation rowMutation =\n        RowMutation.create(tableId, getFullDetectNewPartition())\n            .setCell(\n                MetadataTableAdminDao.CF_MISSING_PARTITIONS,\n                ByteString.copyFromUtf8(MetadataTableAdminDao.QUALIFIER_DEFAULT),","sourceCodeStart":770,"sourceCodeEnd":806,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableDao.java#L770-L806","documentation":"MetadataTableDao.readMissingPartitions logs 'Failed to deserialize missingPartitions: {}' when the Java-serialized missing-partitions blob stored in the Bigtable metadata table cannot be deserialized (SerializationException or NullPointerException). The method then returns an empty/default list, so previously recorded missing partitions are treated as absent.","triggerScenarios":"The DEFAULT qualifier cell in the metadata table contains bytes not compatible with the current class's serialVersionUID — e.g. written by a different Beam version, corrupted cell, or empty cell (NPE).","commonSituations":"Upgrading or downgrading Beam between versions where the serialized class (e.g. MissingPartitions wrapper) changed; manually copied/restored metadata tables; truncated rows.","solutions":["Clear the corrupted missingPartitions cell (or row) so it is rewritten by the current connector version.","Run the pipeline with the same Beam version that wrote the metadata to avoid serialVersionUID mismatch.","If the metadata table is unusable, recreate it via MetadataTableAdminDao and let the pipeline rebuild state.","Accept the empty result if missing-partition bookkeeping can be rebuilt — it only tracks how long partitions have been missing."],"exampleFix":"// before\n// metadata cell written by Beam 2.47 read by Beam 2.54 -> SerializationException\n// after\n// run all jobs against the metadata table with one Beam version, or delete the stale cell:\n// delete row key <stream_partition_missing_partitions> from the metadata table, then restart the pipeline","handlingStrategy":"fallback","validationCode":null,"typeGuard":"boolean isDeserializable(byte[] blob) { try { SerializationUtils.deserialize(blob); return true; } catch (Exception e) { return false; } }","tryCatchPattern":"try { missing = SerializationUtils.deserialize(bytes); } catch (SerializationException | NullPointerException e) { LOG.warn(\"Failed to deserialize missingPartitions: {}\", e.toString()); missing = new MissingPartitions(); }","preventionTips":["Keep one Beam version per metadata table to avoid serialVersionUID drift.","Do not copy/restore metadata tables across versions.","Clear stale cells after upgrades so they are rewritten in the current format."],"tags":["java","apache-beam","bigtable","serialization"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}