apache/druid · error · IOException (IOE)

Unexpected subdirectory [%s]

Error message

Unexpected subdirectory [%s]

What it means

Unzipped segment directories are expected to be flat: pushNoZip uploads each file in indexFilesDir but refuses subdirectories, since nested files cannot be mapped onto the flat S3 key layout for unzipped segments.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentPusher.java:138

        size += file.length();

        try {
          S3Utils.retryS3Operation(
              () -> {
                S3Utils.uploadFileIfPossible(s3Client, config.getDisableAcl(), config.getBucket(), s3Path + file.getName(), file);
                return null;
              }
          );
        }
        catch (S3Exception e) {
          throw handlePushServiceException(e, file.length());
        }
        catch (Exception e) {
          throw new RuntimeException(e);
        }
      } else {
        // Segment directories are expected to be flat.
        throw new IOE("Unexpected subdirectory [%s]", file.getName());
      }
    }

    final int binaryVersion = SegmentUtils.getVersionFromDir(indexFilesDir);
    // V10 unzipped is rangeable: a single druid.segment with a range-readable header. V9 unzipped is a directory of
    // separate smoosh files the range-read path can't consume.
    final boolean rangeable = binaryVersion == IndexIO.V10_VERSION;
    return baseSegment.withSize(size)
                      .withLoadSpec(makeLoadSpec(config.getBucket(), s3Path, rangeable))
                      .withBinaryVersion(binaryVersion);
  }

  @Override
  public Map<String, Object> makeLoadSpec(URI finalIndexZipFilePath)
  {
    // remove the leading "/"
    return makeLoadSpec(finalIndexZipFilePath.getHost(), finalIndexZipFilePath.getPath().substring(1));
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Flatten the segment directory so it contains only files.
2. Re-generate the segment with a current Druid version producing V10 (rangeable druid.segment) layout.
3. Remove stray temp/checkpoint subdirectories before pushing.
4. Zip the segment instead if the flat layout cannot be produced.

Example fix

// before
segmentDir/
  00000.smoosh
  legacy/extra.smoosh   <- causes error
// after
segmentDir/
  00000.smoosh
  00001.smoosh

File[] entries = dir.listFiles();
for (File f : entries) { if (f.isDirectory()) throw new IllegalStateException("nested dir: " + f); }

When it happens

Trigger: Pushing an unzipped segment whose directory contains a subdirectory (e.g. old V9-style multi-smoosh layout or stray temp dirs inside the segment directory).

Common situations: Legacy V9 segments being re-pushed with the no-zip path; leftover hadoop job temp subdirectories inside the segment output; manually assembled segment dirs with nested structure.

Related errors

• An external S3 table with a format must also provide the cor
• Provide either the %s property, or one of the S3 input sourc
• The %s property cannot be provided when the %s property is s
• S3 requires one of %s, %s or %s
• When using the %s parameter, %s must also be provided

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a5573e0c02f2effd. Report an issue: GitHub.