apache/druid · error · IllegalArgumentException

New data cannot be set in buffer till all the old data has…

Error message

New data cannot be set in buffer till all the old data has been read

What it means

SettableByteEntity wraps a ByteBufferInputStream delegate for reading data pushed into the entity. setBuffer replaces the buffer with new data; if the previous buffer still has unread bytes (available() > 0), replacing it would silently discard data, so an IAE is thrown.

Solutions

1. Ensure the consumer fully reads (or explicitly closes/resets) the current buffer before pushing new data
2. Check the consuming task for exceptions or back-pressure causing unread data
3. Upgrade Druid — later versions reworked this input-entity handling for seekable streams

Example fix

// before
entity.setBuffer(nextChunk); // throws if old data remains
// after
if (entity.available() == 0) {
  entity.setBuffer(nextChunk);
}

if (entity.available() > 0) {
  throw new IllegalStateException("Previous buffer not fully consumed; drain before setting new buffer");
}

try {
  entity.setBuffer(newBuffer);
} catch (IllegalArgumentException e) {
  // drain remaining bytes or reset entity, then retry
  log.warn(e, "Buffer not drained; resetting input entity");
  entity.reset();
  entity.setBuffer(newBuffer);
}

When it happens

Trigger: Calling setBuffer (via the entity's open path, e.g. a new chunk arriving from Kafka/Kinesis fetch) while the consumer has not fully read the previous buffer.

Common situations: Slow or stalled consumers in seekable-stream input entities; a producer pushing the next chunk before the row parser finished the previous one; downstream exceptions leaving the stream partially consumed.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Exception while seeking to the
• Failed to load next batch of results
• Failed to stream logs from
• Failed to stream logs from
• IOException wrapping underlying cause

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/6a2c6bed462b62ef. Report an issue: GitHub.

Appendix: source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SettableByteEntity.java:81

  @Override
  public InputStream openRaw()
  {
    // Duplicate the entity buffer, because the stream will update its position.
    final SettableByteBufferInputStream stream = new SettableByteBufferInputStream();
    stream.setBuffer(entity.getBuffer().duplicate());
    return stream;
  }

  public static final class SettableByteBufferInputStream extends InputStream
  {
    @Nullable
    private ByteBufferInputStream delegate;

    public void setBuffer(ByteBuffer newBuffer)
    {
      if (null != delegate && available() > 0) {
        throw new IAE("New data cannot be set in buffer till all the old data has been read");
      }
      this.delegate = new ByteBufferInputStream(newBuffer);
    }

    @Override
    public int read()
    {
      Preconditions.checkNotNull(delegate, "Buffer is not set");
      return delegate.read();
    }

    @Override
    public int read(byte[] bytes, int off, int len)
    {
      Preconditions.checkNotNull(delegate, "Buffer is not set");
      return delegate.read(bytes, off, len);
    }

View on GitHub (pinned to 9b90983fd2)