paperclipai/paperclip · error · Error

Remote file read was truncated for ${remotePath}: ${out.byte

Error message

Remote file read was truncated for ${remotePath}: ${out.byteLength}/${totalBytes} bytes

What it means

Thrown by the readFile method of the command-managed runtime client when the total decoded bytes from the chunked dd|base64 read loop do not equal the byte count reported by the initial 'wc -c' size probe. The loop reads REMOTE_READ_CHUNK_BYTES at a time via dd, base64-encodes each chunk, and accumulates decoded bytes. If the file was truncated, deleted, or modified during the read, or if dd/base64 produced fewer bytes than expected, this integrity check fails.

Source

Thrown at packages/adapter-utils/src/command-managed-runtime.ts:315

      const decodedChunks: Buffer[] = [];
      let decodedSoFar = 0;
      if (totalBytes === 0) {
        await options?.onProgress?.(0, 0);
        return Buffer.alloc(0);
      }
      for (let chunkIndex = 0; decodedSoFar < totalBytes; chunkIndex++) {
        const result = await runShell(
          `dd if=${shellQuote(remotePath)} bs=${REMOTE_READ_CHUNK_BYTES} skip=${chunkIndex} count=1 2>/dev/null | base64`,
        );
        const chunk = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
        if (chunk.byteLength === 0) break;
        decodedChunks.push(chunk);
        decodedSoFar += chunk.byteLength;
        await options?.onProgress?.(Math.min(decodedSoFar, totalBytes), totalBytes);
      }
      const out = Buffer.concat(decodedChunks);
      if (out.byteLength !== totalBytes) {
        throw new Error(`Remote file read was truncated for ${remotePath}: ${out.byteLength}/${totalBytes} bytes`);
      }
      await options?.onProgress?.(out.byteLength, totalBytes);
      return out;
    },
    listFiles: async (remotePath) => {
      const result = await runShell(
        `if [ -d ${shellQuote(remotePath)} ]; then ` +
          `for entry in ${shellQuote(remotePath)}/*; do ` +
          `[ -f "$entry" ] || continue; ` +
          `basename "$entry"; ` +
          `done; ` +
        `fi`,
      );
      return result.stdout
        .split(/\r?\n/)
        .map((entry) => entry.trim())
        .filter((entry) => entry.length > 0)
        .sort((left, right) => left.localeCompare(right));

View on GitHub (pinned to 67001ec6eb)

Solutions

Retry the read if the failure is transient (file being written to concurrently).
Ensure no other process is modifying, truncating, or deleting the file during the read operation.

Example fix

// before: reading a file that is being actively written
await client.readFile('/workspace/app/runtime.log'); // truncated during read

// after: snapshot the file first, then read the snapshot
await client.run(`cp ${shellQuote(remotePath)} ${shellQuote(remotePath + '.snapshot')}`);
const data = await client.readFile(remotePath + '.snapshot');
await client.remove(remotePath + '.snapshot');

Defensive patterns

Strategy: retry

Validate before calling

async function isRemoteFileStable(client: SandboxManagedRuntimeClient, remotePath: string): Promise<boolean> {
  try {
    const size1 = await client.run(`wc -c < ${shellQuote(remotePath)}`);
    const size2 = await client.run(`wc -c < ${shellQuote(remotePath)}`);
    return size1.stdout.trim() === size2.stdout.trim();
  } catch {
    return false;
  }
}

// Call before client.readFile for files that might be actively written:
if (!(await isRemoteFileStable(client, remotePath))) {
  // Snapshot the file first
  await client.run(`cp ${shellQuote(remotePath)} ${shellQuote(remotePath + '.snapshot')}`);
}

Try / catch

try {
  const data = await client.readFile(remotePath);
} catch (error) {
  if (error instanceof Error && error.message.includes('truncated')) {
    // File was modified during read; snapshot it first and retry
    await client.run(`cp ${shellQuote(remotePath)} ${shellQuote(remotePath + '.snapshot')}`);
    const data = await client.readFile(remotePath + '.snapshot');
    await client.remove(remotePath + '.snapshot');
    return data;
  }
  throw error;
}

Prevention

Ensure no other process writes to, truncates, or deletes the file during a readFile operation.
Retry readFile on transient truncation failures—if the file is static, the second read should succeed.
On provider-backed sandboxes with strict resource limits, read smaller files or increase the sandbox's memory/process limits.
Consider using client.run with a single 'cat | base64' for small files to avoid the multi-chunk dd pipeline.

When it happens

Trigger: Calling client.readFile(remotePath) when the file is modified, truncated, or deleted between the 'wc -c' size probe and the chunked dd reads. Also triggered if dd or base64 in the sandbox produces fewer bytes than expected (e.g., dd hitting a read error mid-file, or the sandbox killing the process).

Common situations: Concurrent file writes or truncation during a read (TOCTOU race). The sandbox has resource limits that kill long-running dd/base64 pipelines. The file is on a network filesystem with intermittent failures. A large file read where one chunk fails silently. The file is a log being actively rotated/truncated.

Related errors

AI-assisted analysis of paperclipai/paperclip@67001ec6eb (2026-08-12). Data as JSON: /api/errors/881d00ee4bba2b58. Report an issue: GitHub.