apache/beam · error

OffsetRetainer: failed to save offset to

Error message

OffsetRetainer: failed to save offset to {}. The offset will be lost if the pipeline restarts.

What it means

WARN logged by the Debezium FileSystemOffsetRetainer when atomically persisting the connector offset (write to .tmp then rename) fails. The pipeline deliberately continues running from the in-memory offset, but as the message states, that offset is lost on restart — the connector will resume from the last successfully saved offset and may replay or skip changes accordingly.

Solutions

1. Check the underlying I/O error: permissions, disk full, or an unreachable offset-storage path
2. Ensure the offset directory is writable and durable across pipeline restarts
3. Verify the last saved offset after fixing storage, since the newest offset may have been lost

When it happens

Trigger: Thrown at sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/FileSystemOffsetRetainer.java:159 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8f02d8ebde428fa7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/FileSystemOffsetRetainer.java:159

  public void saveOffset(Map<String, Object> offset) {
    if (offset.equals(lastSavedOffset)) {
      LOG.debug("OffsetRetainer: offset unchanged, skipping write to {}", path);
      return;
    }
    String tmpPath = path + ".tmp";
    try {
      ResourceId tmpResourceId = FileSystems.matchNewResource(tmpPath, /* isDirectory= */ false);
      try (WritableByteChannel channel = FileSystems.create(tmpResourceId, "application/json");
          OutputStream stream = Channels.newOutputStream(channel)) {
        mapper().writeValue(stream, offset);
      }
      ResourceId finalResourceId = FileSystems.matchNewResource(path, /* isDirectory= */ false);
      FileSystems.rename(
          Collections.singletonList(tmpResourceId), Collections.singletonList(finalResourceId));
      lastSavedOffset = offset;
      LOG.debug("OffsetRetainer: saved offset to {}: {}", path, offset);
    } catch (IOException e) {
      LOG.warn(
          "OffsetRetainer: failed to save offset to {}."
              + " The offset will be lost if the pipeline restarts.",
          path,
          e);
    }
  }
}

View on GitHub (pinned to 12126d8942)