{"record":{"id":"893706e7c4088e2d","repo":"apache/hadoop","slug":"change-reported-by-s3-during-s-at-position-s-s","errorCode":null,"errorMessage":"Change reported by S3 during %s at position %s. %s %s was unavailable","messagePattern":"Change reported by S3 during (.+?) at position (.+?)\\. (.+?) (.+?) was unavailable","errorType":"exception","errorClass":"RemoteFileChangedException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ChangeTracker.java","lineNumber":179,"sourceCode":"  /**\n   * Process the response from the server for validation against the\n   * change policy.\n   * @param object object returned; may be null.\n   * @param operation operation in progress.\n   * @param pos offset of read\n   * @throws PathIOException raised on failure\n   * @throws RemoteFileChangedException if the remote file has changed.\n   */\n  public void processResponse(final GetObjectResponse object,\n      final String operation,\n      final long pos) throws PathIOException {\n    if (object == null) {\n      // no object returned. Either mismatch or something odd.\n      if (revisionId != null) {\n        // the requirements of the change detection policy wasn't met: the\n        // object was not returned.\n        versionMismatches.versionMismatchError();\n        throw new RemoteFileChangedException(uri, operation,\n            String.format(CHANGE_REPORTED_BY_S3\n                    + \" during %s\"\n                    + \" at position %s.\"\n                    + \" %s %s was unavailable\",\n                operation,\n                pos,\n                getSource(),\n                getRevisionId()));\n      } else {\n        throw new PathIOException(uri, \"No data returned from GET request\");\n      }\n    }\n\n    processMetadata(object, operation);\n  }\n\n  /**\n   * Process the response from the server for validation against the","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ChangeTracker.java#L161-L197","documentation":"RemoteFileChangedException from ChangeTracker.processResponse when change detection expected a specific revision (revisionId != null, e.g. an etag or version id from the original open/list) but the GET returned no object at all. The tracker reads the source (etag or versionId per fs.s3a.change.detection.policy) on open and re-validates every response; a null object means S3 could not serve the requested revision -- the classic signal that the object was overwritten or deleted while being read. It also bumps the versionMismatches statistic before throwing.","triggerScenarios":"Long reads (DistCP, split reads, commit-time verification) where the object is overwritten/deleted between the initial LIST/open and a later ranged GET; requesting a specific version on a bucket where that version no longer resolves (versioning suspended, or the proxy/gateway strips version parameters).","commonSituations":"Two writers publish to the same key concurrently; a compaction/ETL job rewrites files while a reader still has them open; S3Guard/commit workloads reading through caches or gateways that drop version headers.","solutions":["Rerun the read/job against the current object state -- the old revision's bytes are gone by the time this throws","Eliminate concurrent writers to the same key (unique names, staging committers, or a write-once key layout)","For workloads that tolerate mid-read changes, relax fs.s3a.change.detection.policy (e.g. 'warn') so mismatches log instead of failing","Keep bucket versioning enabled so requested version ids remain resolvable and diagnose which writer replaced the object"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  tracker.processResponse(object, operation, pos);\n} catch (RemoteFileChangedException e) {\n  // the object was overwritten/deleted mid-read: abort this read and restart\n  // from a fresh open() of the current object state\n  LOG.warn(\"Object changed during read, restarting: {}\", uri, e);\n  throw new RetriableException(e);\n}","preventionTips":["Write each key once per run: unique output names or staging committers instead of in-place overwrites","Enable bucket versioning so requested revision ids stay resolvable during long reads","Track the s3a_change_tracking / version mismatch filesystem metrics to detect chronic concurrent-write patterns"],"tags":["s3a","change-detection","consistency","concurrent-write"],"backgroundTag":"remote-file-changed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}