apache/druid · error · SegmentLoadingException

Unable to move segment[%s]: [%s]

Error message

Unable to move segment[%s]: [%s]

What it means

Wrapper around any S3Exception raised while performing the actual copy+delete that relocates a segment to the target bucket/key. It carries the segment id and SDK exception so operators can see why the S3 move failed.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentMover.java:142

                          Maps.filterKeys(
                              loadSpec,
                              new Predicate<>()
                              {
                                @Override
                                public boolean apply(String input)
                                {
                                  return !("bucket".equals(input) || "key".equals(input));
                                }
                              }
                          )
                      )
                      .put("bucket", targetS3Bucket)
                      .put("key", targetS3Path)
                      .build()
      );
    }
    catch (S3Exception e) {
      throw new SegmentLoadingException(e, "Unable to move segment[%s]: [%s]", segment.getId(), e);
    }
  }

  private void safeMove(
      final String s3Bucket,
      final String s3Path,
      final String targetS3Bucket,
      final String targetS3Path
  ) throws SegmentLoadingException
  {
    try {
      S3Utils.retryS3Operation(
          () -> {
            final String copyMsg = StringUtils.format(
                "[s3://%s/%s] to [s3://%s/%s]",
                s3Bucket,
                s3Path,
                targetS3Bucket,

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Read the wrapped S3Exception message/status for the exact S3 error code.
2. Grant s3:PutObject on the target prefix and s3:DeleteObject on the source prefix.
3. Match KMS keys/encryption across buckets if SSE is used, or allow cross-key use via key policies.
4. Retry after throttling errors; reduce move batch size.
5. Verify source and target buckets are in the region(s) the client can reach.

Example fix

// IAM: allow both directions
// after: "Action": ["s3:GetObject","s3:PutObject","s3:DeleteObject"], "Resource": ["arn:aws:s3:::src/*","arn:aws:s3:::dst/*"]

// pre-flight: confirm write access on target
s3Client.headBucket(HeadBucketRequest.builder().bucket(targetBucket).build());
// and verify IAM allows s3:PutObject on target prefix via policy simulator

try { mover.move(segment, target); } catch (SegmentLoadingException e) {
  if (e.getCause() instanceof S3Exception s3e && s3e.statusCode() == 503) retryWithBackoff();
  else throw e;
}

When it happens

Trigger: After validations pass, safeMove -> moveObject issues copyObject/deleteObject; an S3Exception from the SDK (permissions, throttling, KMS, source gone with no fallback) is caught and rethrown here.

Common situations: IAM lacking s3:PutObject on target or s3:DeleteObject on source, cross-region copy denied (CORS/KMS), S3 503 under load, source object deleted concurrently.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Couldn't kill segment[%s]: [%s]
• Failed to get object summaries from S3 bucket[%s], prefix[%s
• Couldn't delete segments from S3. See the task logs for more
• Target S3 bucket is not specified
• Target S3 baseKey is not specified

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/aefa1a448f4e76d3. Report an issue: GitHub.