jackwener/OpenCLI · error · CommandExecutionError

Short read on part ${partNumber}: expected ${chunkSize} byte

Error message

Short read on part ${partNumber}: expected ${chunkSize} bytes, got ${bytesRead}

What it means

While reading each PART_SIZE chunk of the video, tosUpload compares fs.readSync's returned byte count with the expected chunkSize; this CommandExecutionError is thrown on a short read, since uploading a partial buffer would corrupt the part's CRC32 and data.

Source

Thrown at clis/douyin/_shared/tos-upload.js:313

    // Calculate total parts
    const totalParts = Math.ceil(fileSize / PART_SIZE);
    // Track uploaded bytes for progress
    let uploadedBytes = completedParts.length * PART_SIZE;
    if (onProgress)
        onProgress(Math.min(uploadedBytes, fileSize), fileSize);
    const fd = fs.openSync(filePath, 'r');
    try {
        for (let partNumber = 1; partNumber <= totalParts; partNumber++) {
            if (completedPartNumbers.has(partNumber)) {
                continue; // already uploaded
            }
            const offset = (partNumber - 1) * PART_SIZE;
            const chunkSize = Math.min(PART_SIZE, fileSize - offset);
            const buffer = Buffer.allocUnsafe(chunkSize);
            const readFn = _readSyncOverride ?? fs.readSync;
            const bytesRead = readFn(fd, buffer, 0, chunkSize, offset);
            if (bytesRead !== chunkSize) {
                throw new CommandExecutionError(`Short read on part ${partNumber}: expected ${chunkSize} bytes, got ${bytesRead}`);
            }
            const crc32 = await uploadPart(tosUrl, partNumber, uploadId, buffer, auth, uploadHeader, userId);
            completedParts.push({ partNumber, crc32 });
            saveResumeState(resumePath, { uploadId, fileSize, parts: completedParts });
            uploadedBytes = Math.min(offset + chunkSize, fileSize);
            if (onProgress)
                onProgress(uploadedBytes, fileSize);
        }
    }
    finally {
        fs.closeSync(fd);
    }
    const completedKey = await completeMultipartUpload(tosUrl, uploadId, completedParts, auth, uploadHeader, userId);
    deleteResumeState(resumePath);
    return completedKey;
}
// ── Internal exports for testing ─────────────────────────────────────────────
export { PART_SIZE, RESUME_DIR, extractRegionFromHost, getResumeFilePath, loadResumeState, saveResumeState, deleteResumeState, computeAws4Headers, extractUploadId, crc32Hex, gatewayBaseUrl, gatewayHeaders, };

View on GitHub (pinned to 49907e53dc)

Solutions

Ensure the file isn't modified or deleted during upload (pause writers, copy to stable location)
Retry the upload; transient filesystem errors often resolve
Re-run fs.statSync to confirm size matches the resumeState fileSize
If using setReadSyncOverride in tests, make it return the requested byte count

Example fix

// before
const bytesRead = fs.readSync(fd, buffer, 0, chunkSize, offset);
// after (loop until full read)
let bytesRead = 0;
while (bytesRead < chunkSize) {
  const n = fs.readSync(fd, buffer, bytesRead, chunkSize - bytesRead, offset + bytesRead);
  if (n <= 0) break;
  bytesRead += n;
}

Defensive patterns

Strategy: validation

Validate before calling

const { size: sizeNow } = fs.statSync(filePath);
if (sizeNow !== expectedSize) {
  throw new Error(`file changed: expected ${expectedSize}, now ${sizeNow}`);
}

Try / catch

try { await tosUpload(options); }
catch (e) {
  if (String(e.message).startsWith('Short read on part')) {
    console.error('file changed during upload or fs error; retry with the file held stable');
  }
  throw e;
}

Prevention

Never write to/delete the video while it is uploading
Upload from local disk rather than flaky network mounts when possible
Remove custom _readSyncOverride in production paths
Verify resume-state fileSize still matches the current file size

When it happens

Trigger: readFn(fd, buffer, 0, chunkSize, offset) returns fewer bytes than chunkSize — file shrank between stat and read, I/O error, or a testing override (_readSyncOverride) misbehaving.

Common situations: File modified/deleted while the upload is in progress; NFS/network filesystem partial reads; custom read override in tests returning wrong counts.

Related errors

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/01e4f181b63b5af9. Report an issue: GitHub.