rustfs/rustfs · error · io::Error

erasure reader workspace missing

Error message

erasure reader workspace missing

What it means

Internal invariant of ErasureReader::fill_worker_tx: the decode workspace is taken from an Option when the fill worker spawns; BrokenPipe is reported if a spawn is needed but the workspace is gone (source and engine are restored first). Reaching it means the reader was already drained of its components by a previous worker — reuse past the reader's lifetime.

Source

Thrown at crates/ecstore/src/erasure/coding/decode_reader.rs:235

    fn extend_reusable_output_bufs(&mut self, bufs: Vec<Vec<u8>>) {
        for buf in bufs {
            self.push_reusable_output_buf(buf);
        }
    }

    fn fill_worker_tx(&mut self) -> io::Result<mpsc::Sender<FillRequest>> {
        if self.worker.is_none() {
            let Some(source) = self.source.take() else {
                return Err(io::Error::new(ErrorKind::BrokenPipe, "erasure reader source missing"));
            };
            let Some(engine) = self.engine.take() else {
                self.source = Some(source);
                return Err(io::Error::new(ErrorKind::BrokenPipe, "erasure reader engine missing"));
            };
            let Some(workspace) = self.workspace.take() else {
                self.source = Some(source);
                self.engine = Some(engine);
                return Err(io::Error::new(ErrorKind::BrokenPipe, "erasure reader workspace missing"));
            };

            let (tx, rx) = mpsc::channel(1);
            rustfs_io_metrics::record_get_object_fill_worker_started(self.metrics_path, self.fill_policy.as_str());
            let task = tokio::spawn(run_fill_worker(
                source,
                engine,
                workspace,
                self.fill_policy,
                self.metrics_path,
                self.stage_metrics_enabled,
                rx,
            ));
            self.worker = Some(FillWorker { tx, task });
        }

        self.worker
            .as_ref()

View on GitHub (pinned to 9e6e02ea09)

Solutions

1. Scope each ErasureReader to a single GET stream; recreate on retry
2. Propagate terminal errors and drop the reader immediately
3. Report as a bug if it occurs without reader reuse

fn is_reader_workspace_missing(err: &std::io::Error) -> bool {
    err.kind() == std::io::ErrorKind::BrokenPipe
        && err.to_string().contains("workspace missing")
}

match reader.read(&mut buf).await {
    Err(e) if is_reader_workspace_missing(&e) => {
        // reader state consumed: create a new ErasureReader for the next attempt
    }
    r => r,
}

When it happens

Trigger: A fill poll on a reader whose workspace was moved into an earlier, now-terminated worker — reading after a terminal error or after the stream finished.

Common situations: Retry loops reusing the reader; keeping readers in pools/registers beyond one object stream.

Related errors

• erasure reader source missing
• erasure reader engine missing
• erasure reader fill worker missing
• erasure reader fill worker request queue is closed or full
• erasure reader fill future missing

AI-assisted analysis of rustfs/rustfs@9e6e02ea09 (2026-08-16). Data as JSON: /api/errors/fa44d70971f8044a. Report an issue: GitHub.