elastic/elasticsearch · error · IllegalArgumentException

failure store document has unexpected structure, missing req

Error message

failure store document has unexpected structure, missing required [document] field

What it means

Thrown by RecoverFailureDocumentProcessor.execute when the failure-store document lacks the required 'document' field. The processor unpacks a failure-store entry that must contain document, source, and error sub-fields; absence of 'document' means the input is not a valid failure document. IllegalArgumentException using MISSING_DOCUMENT_ERROR_MSG constant.

Source

Thrown at modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RecoverFailureDocumentProcessor.java:54

    public static final String ERROR_FIELD = "error";

    public static final String MISSING_DOCUMENT_ERROR_MSG =
        "failure store document has unexpected structure, missing required [document] field";
    public static final String MISSING_SOURCE_ERROR_MSG =
        "failure store document has unexpected structure, missing required [document.source] field";
    public static final String MISSING_ERROR_ERROR_MSG = "failure store document has unexpected structure, missing required [error] field";

    public static final String TYPE = "recover_failure_document";

    RecoverFailureDocumentProcessor(String tag, String description) {
        super(tag, description);
    }

    @Override
    @SuppressWarnings("unchecked")
    public IngestDocument execute(IngestDocument document) throws Exception {
        if (document.hasField(DOCUMENT_FIELD) == false) {
            throw new IllegalArgumentException(MISSING_DOCUMENT_ERROR_MSG);
        }

        if (document.hasField(SOURCE_FIELD_PATH) == false) {
            throw new IllegalArgumentException(MISSING_SOURCE_ERROR_MSG);
        }

        if (document.hasField(ERROR_FIELD) == false) {
            throw new IllegalArgumentException(MISSING_ERROR_ERROR_MSG);
        }

        // store pre-recovery data in ingest metadata
        storePreRecoveryData(document);

        // Get the nested 'document' field, which holds the original document and metadata.
        Map<String, Object> failedDocument = (Map<String, Object>) document.getFieldValue(DOCUMENT_FIELD, Map.class);

        // Copy the original index, routing, and id back to the document's metadata.
        String originalIndex = (String) failedDocument.get(INDEX_FIELD);

View on GitHub (pinned to db6a809a66)

Solutions

Only route genuine failure-store documents to the recover_failure_document processor.
Verify the failure store schema has document/source/error fields (use GET on a sample failure doc to inspect).
If ingesting non-failure data, remove the recover_failure_document processor from that pipeline.

Example fix

// before
{"recover_failure_document": {}}
// applied to a regular doc with no 'document' field
// after - gate the processor:
{"recover_failure_document": {"if": "ctx.containsKey('document') && ctx.containsKey('error')"}}

Defensive patterns

Strategy: type-guard

Validate before calling

if (!doc.hasField("document")) {
    // do not route this document through recover_failure_document
}

Type guard

static boolean isFailureStoreEnvelope(IngestDocument doc) {
    return doc.hasField("document") && doc.hasField("source") && doc.hasField("error");
}

Try / catch

try {
    recoverProcessor.execute(doc);
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("missing required [document] field")) {
        // skip — not a failure-store document
    } else throw e;
}

Prevention

Only route genuine failure-store documents to recover_failure_document.
Add a conditional checking for the document/error/source fields before the processor.
Inspect a sample failure document to confirm the expected envelope schema.

When it happens

Trigger: Recover failure processor invoked on a document that does not have the failure-store envelope structure — specifically missing the 'document' key.

Common situations: Pipeline misconfiguration running recover_failure_document on regular (non-failure-store) documents; failure store schema changed between versions; manually-crafted failure documents missing fields.

Related errors

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/e21cb61264828bd5. Report an issue: GitHub.