{"record":{"id":"0349e7c136973899","repo":"nautechsystems/nautilus_trader","slug":"execution-payload-rewrap-left-remaining-row-s","errorCode":null,"errorMessage":"Execution payload rewrap left {remaining} row(s)","messagePattern":"Execution payload rewrap left (.+?) row\\(s\\)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":5470,"sourceCode":"        .await\n        .context(\"failed to load execution payload rewrap batch\")?;\n\n        if rows.is_empty() {\n            sqlx::query(\"LOCK TABLE execution_transaction_hash IN SHARE ROW EXCLUSIVE MODE\")\n                .execute(&mut *transaction)\n                .await\n                .context(\"failed to lock execution payload rewrap completion\")?;\n            let remaining = sqlx::query_scalar::<_, i64>(\n                \"SELECT COUNT(*) FROM execution_transaction_hash \\\n                 WHERE payload_expected \\\n                   AND (raw_transaction IS NOT NULL OR sealed_transaction IS NULL \\\n                        OR substring(sealed_transaction FROM 2 FOR 32) <> $1)\",\n            )\n            .bind(keys.active_key_id().as_slice())\n            .fetch_one(&mut *transaction)\n            .await\n            .context(\"failed to verify execution payload rewrap completion\")?;\n            anyhow::ensure!(\n                remaining == 0,\n                \"Execution payload rewrap left {remaining} row(s)\"\n            );\n            sqlx::query(\n                \"UPDATE execution_payload_state SET operation = 'ready', updated_at = NOW() \\\n                 WHERE component = 'signed_transactions' AND operation = 'rewrap'\",\n            )\n            .execute(&mut *transaction)\n            .await\n            .context(\"failed to mark execution payload rewrap complete\")?;\n            transaction\n                .commit()\n                .await\n                .context(\"failed to commit execution payload rewrap completion\")?;\n            return Ok(true);\n        }\n\n        for hash in rows {","sourceCodeStart":5452,"sourceCodeEnd":5488,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L5452-L5488","documentation":"When a rewrap batch finds no more rows to process, the code locks the table and runs a verification COUNT of rows that still have `payload_expected` with a NULL/missing plaintext constraint violation: raw_transaction set, sealed_transaction NULL, or an envelope not sealed under the new active key. If any rows remain (`remaining != 0`), the rewrap is not actually complete and this error aborts the state transition to 'ready'. It is a safety check that the batch loop truly finished the rewrap.","triggerScenarios":"Rows exist that the batch SELECT (which filters on `substring(sealed_transaction FROM 2 FOR 32) <> $1` only for non-null envelopes) skipped but the stricter verification COUNT flags — e.g. rows with `payload_expected=true` but `sealed_transaction IS NULL` or `raw_transaction IS NOT NULL` (plaintext rows), or rows inserted concurrently between the last batch and the table lock on a table where the lock was not held by a competing writer.","commonSituations":"A writer inserted new signed-transaction rows with plaintext `raw_transaction` while a rewrap was running; a previous failed rewrap left rows half-migrated (NULL envelopes); manual data loads backfilled `payload_expected` rows without sealed envelopes; trigger/ETL jobs writing to `execution_transaction_hash` during maintenance.","solutions":["Identify the leftover rows: `SELECT id FROM execution_transaction_hash WHERE payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NULL OR substring(sealed_transaction FROM 2 FOR 32) <> $1)`.","Stop all writers to `execution_transaction_hash` (adapter instances, ETL jobs) and re-run the rewrap so the batch loop can process the remaining rows.","Fix half-migrated rows: either re-seal them under the active key (restore from a source with plaintext + re-seal) or correct `payload_expected` if these rows should not carry payloads.","Check for concurrent insert paths and ensure they seal payloads with the current active key before the rewrap declares completion."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"let remaining: i64 = sqlx::query_scalar(\"SELECT COUNT(*) FROM execution_transaction_hash WHERE payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NULL OR substring(sealed_transaction FROM 2 FOR 32) <> $1)\")\n    .bind(active_key_id.as_slice()).fetch_one(&mut *tx).await?;\n// only declare the rewrap complete when remaining == 0","typeGuard":null,"tryCatchPattern":"match rewrap_result {\n    Ok(done) if done => {/* rewrap complete */},\n    Err(e) if e.to_string().contains(\"Execution payload rewrap left\") => {\n        // quiesce writers, then retry the batch loop after fixing/remediating rows\n    },\n    Err(e) => return Err(e),\n}","preventionTips":["Quiesce all writers to `execution_transaction_hash` for the duration of a rewrap.","Monitor the leftover-row COUNT during maintenance and alert if it stops decreasing.","Ensure all insert paths seal payloads with the current active key before marking `payload_expected`.","After any restore/backfill, run a full rewrap pass before declaring the storage 'ready'."],"tags":["database","data-integrity","encryption","postgres","rust"],"backgroundTag":"database-query-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}