elastic/elasticsearch · error · IllegalArgumentException

failure store document has unexpected structure, missing req

Error message

failure store document has unexpected structure, missing required [error] field

What it means

Thrown by RecoverFailureDocumentProcessor.execute when the IngestDocument lacks the 'error' field. The processor needs the 'error' metadata (along with 'document' and 'document.source') to fully reconstruct a failed document for reprocessing.

Source

Thrown at modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RecoverFailureDocumentProcessor.java:62

    public static final String TYPE = "recover_failure_document";

    RecoverFailureDocumentProcessor(String tag, String description) {
        super(tag, description);
    }

    @Override
    @SuppressWarnings("unchecked")
    public IngestDocument execute(IngestDocument document) throws Exception {
        if (document.hasField(DOCUMENT_FIELD) == false) {
            throw new IllegalArgumentException(MISSING_DOCUMENT_ERROR_MSG);
        }

        if (document.hasField(SOURCE_FIELD_PATH) == false) {
            throw new IllegalArgumentException(MISSING_SOURCE_ERROR_MSG);
        }

        if (document.hasField(ERROR_FIELD) == false) {
            throw new IllegalArgumentException(MISSING_ERROR_ERROR_MSG);
        }

        // store pre-recovery data in ingest metadata
        storePreRecoveryData(document);

        // Get the nested 'document' field, which holds the original document and metadata.
        Map<String, Object> failedDocument = (Map<String, Object>) document.getFieldValue(DOCUMENT_FIELD, Map.class);

        // Copy the original index, routing, and id back to the document's metadata.
        String originalIndex = (String) failedDocument.get(INDEX_FIELD);
        if (originalIndex != null) {
            document.setFieldValue(IngestDocument.Metadata.INDEX.getFieldName(), originalIndex);
        }

        String originalRouting = (String) failedDocument.get(ROUTING_FIELD);
        if (originalRouting != null) {
            document.setFieldValue(IngestDocument.Metadata.ROUTING.getFieldName(), originalRouting);
        }

View on GitHub (pinned to db6a809a66)

Solutions

1. Confirm the failure-store entry retains its 'error' field through any intermediate processing.
2. Avoid stripping the 'error' key in upstream processors when this recovery processor runs downstream.
3. Validate the three required fields (document, document.source, error) before submitting to the pipeline.

Example fix

// before: error field removed by an upstream script processor
// after: preserve the error field
if (ingestDocument.hasField("error") == false) {
    throw new IllegalStateException("failure-store entry missing error metadata");
}

// Verify the 'error' field is present before recovery
if (document.hasField("error") == false) {
    // not a complete failure-store entry; do not run recovery
    return;
}

boolean hasErrorMetadata(IngestDocument doc) {
    return doc.hasField("error");
}

try {
    // run pipeline
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("missing required [error]")) {
        // route to dead-letter; failure-store entry is incomplete
    } else { throw e; }
}

When it happens

Trigger: Running the recover_failure_document processor on a document that has 'document' and 'document.source' but is missing the top-level 'error' key describing the original failure.

Common situations: Partial failure-store entries where the error metadata was stripped or never written. Custom failure pipelines that drop the error field before recovery. Version mismatch between failure-store writer and reader.

Related errors

• failure store document has unexpected structure, missing req
• unable to set domain information for document
• field [{}] doesn't exist
• field [{}] already exists
• invalid data stream name: [{}]; must follow naming scheme <t

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/5d886e19d543b9c6. Report an issue: GitHub.