apache/druid · warning

failed on syncing fd

Error message

failed on syncing fd [%d], offset [%d], bytes [%d], ret_code [%d], errno [%d]

What it means

NativeIO.trySyncFileRange calls Linux's sync_file_range(2) via JNA to asynchronously flush a file's dirty pages to disk. A non-zero return code means the kernel rejected the sync; the code logs a warning with ret_code and errno and returns without throwing, since syncing is a best-effort optimization. Unsupported platforms set syncFileRangePossible=false permanently after the first failure.

Solutions

1. Check the logged errno: EINVAL typically means the filesystem (overlayfs, NFS, tmpfs) doesn't support sync_file_range
2. Move the data directory to a supported filesystem (ext4/xfs on a real block device)
3. Ensure fd/offset/length are valid and the file is open when sync is attempted
4. Treat as advisory: data correctness is unaffected; the call is skipped after repeated unsupported errors

Example fix

// before: bind-mounting Druid segment cache onto overlayfs/tmpfs in Docker
VOLUME ["/opt/druid/var"]
// after: use a volume on ext4/xfs
// docker run -v druid-data:/opt/druid/var ...  # volume backed by block device

// Only rely on sync_file_range on Linux with a supported local filesystem
boolean syncSupported = System.getProperty("os.name").equals("Linux")
    && !mountPoint.startsWith("/mnt") /* exclude NFS/overlay mounts */;

// Best-effort sync: log and continue
try {
  NativeIO.trySyncFileRange(fd, offset, len, flags);
} catch (UnsupportedOperationException e) {
  // platform/filesystem unsupported; skip syncing permanently
}

When it happens

Trigger: Calling trySyncFileRange on a file descriptor where sync_file_range returns an error: invalid fd/offset/length alignment on some filesystems, EINVAL on unsupported filesystems (e.g. NFS, tmpfs, overlayfs), or EBADF/EIO on a closed or errored fd.

Common situations: Running inside containers with overlayfs backing stores where sync_file_range is unsupported; writing to NFS mounts; using file offsets not aligned to page size on filesystems that require it; kernel versions where the syscall isn't available for the fs.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Unsatisfied Link error: posix_fadvise failed on file…
• Cannot create directory
• Cannot create tempDir
• Cannot delete temp file
• Cannot list contents of

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/76bbf2377adf8bed. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/java/util/common/io/NativeIO.java:177

  }

  /**
   * Sync part of an open file to the file system.
   *
   * @param fd      The file descriptor of the source file.
   * @param offset  The offset within the file.
   * @param nbytes  The number of bytes to be synced.
   * @param flags   Signal how to synchronize
   */
  private static void trySyncFileRange(int fd, long offset, long nbytes, int flags)
  {
    if (!initialized || !syncFileRangePossible || fd < 0) {
      return;
    }
    try {
      int ret_code = sync_file_range(fd, offset, nbytes, flags);
      if (ret_code != 0) {
        log.warn("failed on syncing fd [%d], offset [%d], bytes [%d], ret_code [%d], errno [%d]",
            fd, offset, nbytes, ret_code, Native.getLastError());
        return;
      }
    }
    catch (UnsupportedOperationException uoe) {
      log.warn(uoe, "sync_file_range is not supported");
      syncFileRangePossible = false;
    }
    catch (UnsatisfiedLinkError nle) {
      log.warn(nle, "sync_file_range failed on fd [%d], offset [%d], bytes [%d]", fd, offset, nbytes);
      syncFileRangePossible = false;
    }
    catch (Exception e) {
      log.warn(e, "Unknown exception: sync_file_range failed on fd [%d], offset [%d], bytes [%d]",
          fd, offset, nbytes);
      syncFileRangePossible = false;
    }
  }

View on GitHub (pinned to 9b90983fd2)