apache/druid · error · IllegalStateException

Metadata should be null because BatchAppenderatorDriver…

Error message

Metadata should be null because BatchAppenderatorDriver never persists it

What it means

BatchAppenderatorDriver never persists run metadata (its underlying appenderator's startJob() must therefore return null metadata). If metadata is non-null, an invariant of the batch driver has been violated — likely the appenderator was configured or subclassed to persist metadata — and this ISE is thrown to surface the inconsistency.

Solutions

  1. Ensure BatchAppenderatorDriver wraps a BatchAppenderator (created via Appenderators.createBatch), not the streaming Appenderator
  2. Point the appenderator's base persist directory at a clean location free of prior jobs' persisted state
  3. If stale state exists, remove the old persist directories for this data source before starting the job

Example fix

// before
Appenderator appenderator = Appenderators.createAppendee(...); // streaming appenderator
BatchAppenderatorDriver driver = new BatchAppenderatorDriver(appenderator, ...);
// after
Appenderator appenderator = Appenderators.createBatch(dataSchema, tuningConfig, ...);
BatchAppenderatorDriver driver = new BatchAppenderatorDriver(appenderator, ...);
Defensive patterns

Strategy: try-catch

Validate before calling

Object meta = appenderator.startJob();
if (meta != null) {
  throw new IllegalStateException("BatchAppenderatorDriver requires an appenderator that persists no metadata");
}

Type guard

boolean isBatchAppenderator(Appenderator a) {
  return a instanceof BatchAppenderator;
}

Try / catch

try {
  driver.startJob(lockHelper);
} catch (ISE e) {
  if (e.getMessage().contains("Metadata should be null")) {
    cleanPersistDirAndRecreateDriver();
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling BatchAppenderatorDriver.startJob() when the wrapped appenderator's startJob() returns non-null metadata — e.g. a BatchAppenderator instance that loaded persisted metadata from a prior run, or using a non-batch Appenderator implementation with this driver.

Common situations: Mixing Appenderator implementations: wrapping a regular Appenderator (which persists metadata) in BatchAppenderatorDriver; stale persisted state from a misconfigured base persist directory being picked up at startJob.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f8d917d4d72c634d. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderatorDriver.java:93

  @Nullable
  public Object startJob()
  {
    return startJob(AppenderatorDriverSegmentLockHelper.NOOP);
  }

  /**
   * This method always returns null because batch ingestion doesn't support restoring tasks on failures.
   *
   * @return always null
   */
  @Override
  @Nullable
  public Object startJob(AppenderatorDriverSegmentLockHelper lockHelper)
  {
    final Object metadata = appenderator.startJob();
    if (metadata != null) {
      throw new ISE("Metadata should be null because BatchAppenderatorDriver never persists it");
    }
    return null;
  }

  /**
   * Add a row. Must not be called concurrently from multiple threads.
   *
   * @param row          the row to add
   * @param sequenceName sequenceName for this row's segment
   *
   * @return {@link AppenderatorDriverAddResult}
   *
   * @throws IOException if there is an I/O error while allocating or writing to a segment
   */
  public AppenderatorDriverAddResult add(
      InputRow row,
      String sequenceName
  ) throws IOException

View on GitHub (pinned to 9b90983fd2)