apache/hadoop · error · RemoteFileChangedException

Change reported by S3 during %s at position %s. %s %s was un

Error message

Change reported by S3 during %s at position %s. %s %s was unavailable

What it means

RemoteFileChangedException from ChangeTracker.processResponse when change detection expected a specific revision (revisionId != null, e.g. an etag or version id from the original open/list) but the GET returned no object at all. The tracker reads the source (etag or versionId per fs.s3a.change.detection.policy) on open and re-validates every response; a null object means S3 could not serve the requested revision -- the classic signal that the object was overwritten or deleted while being read. It also bumps the versionMismatches statistic before throwing.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ChangeTracker.java:179

  /**
   * Process the response from the server for validation against the
   * change policy.
   * @param object object returned; may be null.
   * @param operation operation in progress.
   * @param pos offset of read
   * @throws PathIOException raised on failure
   * @throws RemoteFileChangedException if the remote file has changed.
   */
  public void processResponse(final GetObjectResponse object,
      final String operation,
      final long pos) throws PathIOException {
    if (object == null) {
      // no object returned. Either mismatch or something odd.
      if (revisionId != null) {
        // the requirements of the change detection policy wasn't met: the
        // object was not returned.
        versionMismatches.versionMismatchError();
        throw new RemoteFileChangedException(uri, operation,
            String.format(CHANGE_REPORTED_BY_S3
                    + " during %s"
                    + " at position %s."
                    + " %s %s was unavailable",
                operation,
                pos,
                getSource(),
                getRevisionId()));
      } else {
        throw new PathIOException(uri, "No data returned from GET request");
      }
    }

    processMetadata(object, operation);
  }

  /**
   * Process the response from the server for validation against the

View on GitHub (pinned to 2add963021)

Solutions

1. Rerun the read/job against the current object state -- the old revision's bytes are gone by the time this throws
2. Eliminate concurrent writers to the same key (unique names, staging committers, or a write-once key layout)
3. For workloads that tolerate mid-read changes, relax fs.s3a.change.detection.policy (e.g. 'warn') so mismatches log instead of failing
4. Keep bucket versioning enabled so requested version ids remain resolvable and diagnose which writer replaced the object

try {
  tracker.processResponse(object, operation, pos);
} catch (RemoteFileChangedException e) {
  // the object was overwritten/deleted mid-read: abort this read and restart
  // from a fresh open() of the current object state
  LOG.warn("Object changed during read, restarting: {}", uri, e);
  throw new RetriableException(e);
}

When it happens

Trigger: Long reads (DistCP, split reads, commit-time verification) where the object is overwritten/deleted between the initial LIST/open and a later ranged GET; requesting a specific version on a bucket where that version no longer resolves (versioning suspended, or the proxy/gateway strips version parameters).

Common situations: Two writers publish to the same key concurrently; a compaction/ETL job rewrites files while a reader still has them open; S3Guard/commit workloads reading through caches or gateways that drop version headers.

Related errors

• File to rename disappeared during the rename operation.
• No data returned from GET request
• Content changed
• Unsupported block buffer "{name}"
• Block length mismatch, len={len} but r={r}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/893706e7c4088e2d. Report an issue: GitHub.