apache/druid · error · ISE

Can't find overwritingSegmentMeta for interval[%s]

Error message

Can't find overwritingSegmentMeta for interval[%s]

What it means

When using segment locking during replace ingestion, OverlordCoordinatingSegmentAllocator needs the OverwritingRootGenerationPartitions metadata for a row's interval to build a NumberedOverwritePartialShardSpec. The taskLockHelper claims to have overwriting root-generation partitions for that interval, but the lookup returns null — an inconsistent state — so the allocator throws ISE.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/common/task/OverlordCoordinatingSegmentAllocator.java:123

  {
    return internalAllocator.allocate(row, sequenceName, previousSegmentId, skipSegmentLineageCheck);
  }

  private static PartialShardSpec createPartialShardSpec(
      AbstractTask.IngestionMode ingestionMode,
      PartitionsSpec partitionsSpec,
      TaskLockHelper taskLockHelper,
      Interval interval
  )
  {
    if (partitionsSpec.getType() == SecondaryPartitionType.LINEAR) {
      if (taskLockHelper.isUseSegmentLock()) {
        if (taskLockHelper.hasOverwritingRootGenerationPartition(interval) && (ingestionMode
                                                                               != AbstractTask.IngestionMode.APPEND)) {
          final OverwritingRootGenerationPartitions overwritingRootGenerationPartitions = taskLockHelper
              .getOverwritingRootGenerationPartition(interval);
          if (overwritingRootGenerationPartitions == null) {
            throw new ISE("Can't find overwritingSegmentMeta for interval[%s]", interval);
          }
          return new NumberedOverwritePartialShardSpec(
              overwritingRootGenerationPartitions.getStartRootPartitionId(),
              overwritingRootGenerationPartitions.getEndRootPartitionId(),
              overwritingRootGenerationPartitions.getMinorVersionForNewSegments()
          );
        }
      }
      return NumberedPartialShardSpec.instance();
    } else {
      throw new ISE(
          "%s is not supported for partitionsSpec[%s]",
          OverlordCoordinatingSegmentAllocator.class.getName(),
          partitionsSpec.getClass().getName()
      );
    }
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the task type is a proper replace (not append) and that ingestion state registered overwriting partitions for every interval in the spec
Re-submit the task; if it recurs, disable segment locking (segment lock -> time chunk lock) so the overwrite metadata isn't required
Check taskLockHelper initialization/registration code for the failing interval and ensure IngestionMode isn't misconfigured

Defensive patterns

Strategy: try-catch

Validate before calling

if (taskLockHelper.hasOverwritingRootGenerationPartition(interval)
    && taskLockHelper.getOverwritingRootGenerationPartition(interval) == null) { throw new IllegalStateException("overwrite metadata missing for " + interval); }

Try / catch

try { shardSpec = allocator.partialShardSpec(row, seq, prevId, false); } catch (ISE e) { if (e.getMessage().contains("Can't find overwritingSegmentMeta")) { switchToTimeChunkLockAndResubmit(); } else { throw e; } }

Prevention

Ensure replace tasks register overwriting partitions for every spec interval
Avoid concurrent tasks mutating the same interval while replacing
Fall back to time-chunk locking when segment-lock replace metadata is unstable

When it happens

Trigger: partialShardSpec is called for an interval where hasOverwritingRootGenerationPartition(interval) is true but getOverwritingRootGenerationPartition(interval) returns null, typically when the overwriting metadata was cleared/never registered (e.g. non-APPEND replace task racing with helper updates).

Common situations: Replace-type ingestion tasks with useSegmentLock enabled whose lock helper state wasn't populated for all intervals; concurrent tasks modifying the interval map; task restarted/resumed losing in-memory helper state.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ec160a6d1f5154e5. Report an issue: GitHub.