apache/beam · warning

Failed to deserialize missingPartitions

Error message

Failed to deserialize missingPartitions: {}

What it means

MetadataTableDao.readMissingPartitions logs 'Failed to deserialize missingPartitions: {}' when the Java-serialized missing-partitions blob stored in the Bigtable metadata table cannot be deserialized (SerializationException or NullPointerException). The method then returns an empty/default list, so previously recorded missing partitions are treated as absent.

Solutions

1. Clear the corrupted missingPartitions cell (or row) so it is rewritten by the current connector version.
2. Run the pipeline with the same Beam version that wrote the metadata to avoid serialVersionUID mismatch.
3. If the metadata table is unusable, recreate it via MetadataTableAdminDao and let the pipeline rebuild state.
4. Accept the empty result if missing-partition bookkeeping can be rebuilt — it only tracks how long partitions have been missing.

Example fix

// before
// metadata cell written by Beam 2.47 read by Beam 2.54 -> SerializationException
// after
// run all jobs against the metadata table with one Beam version, or delete the stale cell:
// delete row key <stream_partition_missing_partitions> from the metadata table, then restart the pipeline

boolean isDeserializable(byte[] blob) { try { SerializationUtils.deserialize(blob); return true; } catch (Exception e) { return false; } }

try { missing = SerializationUtils.deserialize(bytes); } catch (SerializationException | NullPointerException e) { LOG.warn("Failed to deserialize missingPartitions: {}", e.toString()); missing = new MissingPartitions(); }

When it happens

Trigger: The DEFAULT qualifier cell in the metadata table contains bytes not compatible with the current class's serialVersionUID — e.g. written by a different Beam version, corrupted cell, or empty cell (NPE).

Common situations: Upgrading or downgrading Beam between versions where the serialized class (e.g. MissingPartitions wrapper) changed; manually copied/restored metadata tables; truncated rows.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• Cannot decode object from input stream.
• cannot encode a null BitSet
• cannot encode a null byte[]
• cannot encode a null Integer
• cannot encode a null Long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3cc064cb26ee4583. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableDao.java:788

    Row row = dataClient.readRow(tableId, getFullDetectNewPartition(), missingPartitionsFilter);

    if (row == null
        || row.getCells(
                MetadataTableAdminDao.CF_MISSING_PARTITIONS,
                MetadataTableAdminDao.QUALIFIER_DEFAULT)
            .isEmpty()) {
      return missingPartitions;
    }
    ByteString serializedMissingPartition =
        row.getCells(
                MetadataTableAdminDao.CF_MISSING_PARTITIONS,
                MetadataTableAdminDao.QUALIFIER_DEFAULT)
            .get(0)
            .getValue();
    try {
      missingPartitions = SerializationUtils.deserialize(serializedMissingPartition.toByteArray());
    } catch (SerializationException | NullPointerException exception) {
      LOG.warn("Failed to deserialize missingPartitions: {}", exception.toString());
    }
    return missingPartitions;
  }

  /**
   * Write to metadata table serialized missing partitions and how long they have been missing.
   *
   * @param missingPartitionDurations missing partitions and duration.
   */
  public void writeDetectNewPartitionMissingPartitions(
      HashMap<ByteStringRange, Instant> missingPartitionDurations) {
    long nowMicros = Instant.now().getMillis() * 1000L;
    byte[] serializedMissingPartition = SerializationUtils.serialize(missingPartitionDurations);
    RowMutation rowMutation =
        RowMutation.create(tableId, getFullDetectNewPartition())
            .setCell(
                MetadataTableAdminDao.CF_MISSING_PARTITIONS,
                ByteString.copyFromUtf8(MetadataTableAdminDao.QUALIFIER_DEFAULT),

View on GitHub (pinned to 12126d8942)