apache/beam · critical · RuntimeException

OffsetRetainer: failed to read offset from . Delete the file

Error message

OffsetRetainer: failed to read offset from . Delete the file to restart from the beginning.

What it means

FileSystemOffsetRetainer.loadOffset reads the persisted offset file so a resumable Debezium pipeline can restart where it stopped. A FileNotFoundException is benign (start from beginning), but any other IOException while reading is wrapped in this RuntimeException because a corrupt/unreadable offset file must not be silently ignored.

Source

Thrown at sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/FileSystemOffsetRetainer.java:120

   * Reads the offset JSON file and returns its contents, or {@code null} if the file does not yet
   * exist (first run). Throws {@link RuntimeException} if the file exists but cannot be read, to
   * prevent silently reprocessing data from the beginning.
   */
  @Override
  public @Nullable Map<String, Object> loadOffset() {
    try {
      ResourceId resourceId = FileSystems.matchNewResource(path, /* isDirectory= */ false);
      try (ReadableByteChannel channel = FileSystems.open(resourceId);
          InputStream stream = Channels.newInputStream(channel)) {
        Map<String, Object> offset = mapper().readValue(stream, MAP_TYPE);
        LOG.info("OffsetRetainer: loaded offset from {}: {}", path, offset);
        return offset;
      }
    } catch (FileNotFoundException e) {
      LOG.info("OffsetRetainer: no offset file found at {}; starting from the beginning.", path);
      return null;
    } catch (IOException e) {
      throw new RuntimeException(
          "OffsetRetainer: failed to read offset from "
              + path
              + ". "
              + "Delete the file to restart from the beginning.",
          e);
    }
  }

  /**
   * Serialises {@code offset} to JSON and writes it atomically to the configured path.
   *
   * <p>If the offset is identical to the last successfully written one, the write is skipped to
   * avoid unnecessary I/O on every checkpoint.
   *
   * <p>Otherwise the data is first written to a {@code .tmp} sibling file and then renamed to the
   * final path, so a mid-write crash leaves the previous offset intact.
   *
   * <p>Errors are logged as warnings and swallowed so the pipeline continues.

View on GitHub (pinned to 12126d8942)

Solutions

Delete the offset file at the logged path to restart consumption from the beginning (as the message suggests).
Fix filesystem permissions or mount health so the runner's service account can read the file.
If the offset is valuable, inspect/repair the file rather than deleting, then re-run.
Make the retention directory reliable (local persistent disk or healthy GCS/HDFS mount).

Example fix

// shell
rm /var/beam/debezium-offset/offset.json  # then re-launch the pipeline

Defensive patterns

Strategy: fallback

Validate before calling

File offsetFile = new File(path);
if (offsetFile.exists() && !offsetFile.canRead()) {
  LOG.warn("Offset file unreadable; will restart from beginning");
}

Try / catch

try {
  pipeline.run();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("OffsetRetainer: failed to read offset")) {
    new File(offsetPath).delete(); // restart from beginning
    pipeline.run();
  } else throw e;
}

Prevention

Run all pipeline workers under the same service account that owns the offset file
Use a reliable, healthy storage backend for offset retention
Monitor disk health/mount state before resuming jobs
Back up the offset file before infrastructure maintenance

When it happens

Trigger: The offset file at `path` exists but cannot be read: permission denied, I/O error on the filesystem, or a truncated/corrupt file causing the deserializing read to fail with IOException.

Common situations: Shared/NFS mount flakiness, offset file left unreadable after a job ran as a different service account, disk-full or hardware failure during a previous saveOffset.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/87d25b0d82a384f5. Report an issue: GitHub.