apache/druid · error · IllegalStateException

Requested to skip [%s] bytes, but actual number of bytes ski

Error message

Requested to skip [%s] bytes, but actual number of bytes skipped is [%s]

What it means

HttpEntity.openInputStream calls InputStream.skip(offset) to seek to the byte offset of a split within an HTTP response, and verifies the stream actually skipped the requested number of bytes. HTTP streams are not guaranteed to honor skip, so when fewer bytes are skipped the entity closes the stream and fails the split instead of silently reading misaligned data.

Source

Thrown at processing/src/main/java/org/apache/druid/data/input/impl/HttpEntity.java:123

    urlConnection.addRequestProperty(HttpHeaders.RANGE, StringUtils.format("bytes=%d-", offset));
    final String contentRange = urlConnection.getHeaderField(HttpHeaders.CONTENT_RANGE);
    final boolean withContentRange = contentRange != null && contentRange.startsWith("bytes ");
    if (withContentRange && offset > 0) {
      return urlConnection.getInputStream();
    } else {
      if (!withContentRange && offset > 0) {
        LOG.warn(
            "Since the input source doesn't support range requests, the object input stream is opened from the start and "
            + "then skipped. This may make the ingestion speed slower. Consider enabling prefetch if you see this message"
            + " a lot."
        );
      }
      InputStream in = urlConnection.getInputStream();
      try {
        final long skipped = in.skip(offset);
        if (skipped != offset) {
          in.close();
          throw new ISE("Requested to skip [%s] bytes, but actual number of bytes skipped is [%s]", offset, skipped);
        } else {
          return in;
        }
      }
      catch (IOException ex) {
        in.close();
        throw ex;
      }
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Ensure the HTTP server supports Range requests (Accept-Ranges: bytes) and serves the file uncompressed
2. Retry the split; transient truncation often succeeds on a second attempt
3. Serve the data over a range-capable source (S3 with Range support, HDFS, local files) instead of a non-compliant HTTP endpoint
4. Disable compression for this endpoint or use Content-Length-aligned static file serving

// Before relying on offsets, verify the endpoint supports ranges
HttpURLConnection c = (HttpURLConnection) new URI(uri).toURL().openConnection();
c.setRequestMethod("HEAD");
boolean rangeOk = "bytes".equalsIgnoreCase(c.getHeaderField("Accept-Ranges"));

try { return entity.open(); } catch (ISE e) { if (e.getMessage().startsWith("Requested to skip")) { /* re-queue split or switch to range-capable source */ } throw e; }

When it happens

Trigger: Reading a range/offset of an HTTP-served file where the server (or a proxy) does not support byte ranges and returns the body from byte 0, or terminates/truncates the response so skip() advances fewer than 'offset' bytes.

Common situations: Servers or CDNs ignoring the Range header; compressed (gzip) responses where skip cannot jump ahead; flaky proxies cutting responses short; very large offsets on keep-alive connections that reset.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Error occurred while trying to read uri:
• Specify only one of %s or %s
• Exception while getting active tasks from Overlord. Will ret
• OpenLineage event dropped: all [%d] attempts to POST to [%s]
• Error reading response

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/b702be47aa99a2a8. Report an issue: GitHub.