Back Up Large Data With Streaming#

Download a large file and save it to the file system without loading the whole thing into memory. This is the example to copy when the data is bigger than the run’s memory limit and buffering it would fail the run.

The Script#

// backup.js
import fs from "fs";
import env from "env";

const KEEP = 7;
const dir = "/backups";
const stamp = new Date().toISOString().replace(/[:.]/g, "-");
const part = `${dir}/backup-${stamp}.part`;
const target = `${dir}/backup-${stamp}.bak`;

await fs.mkdir(dir, { recursive: true });

const res = await fetch(env.get("BACKUP_URL"));
if (!res.ok) {
	throw new Error(`Download failed: ${res.status}`);
}

// fs.writeFile accepts a ReadableStream, so the response is written to
// disk chunk by chunk and never held whole in memory. The download goes
// to a temporary .part file and only becomes a backup once it finishes.
let bytes = 0;
try {
	await fs.writeFile(part, res.body);
	bytes = Number((await fs.stat(part)).size);
} catch (err) {
	await fs.remove(part);
	throw err;
}

// Commit: a rename is a single transaction, so the backup appears whole
// or not at all, even if the run is killed mid-download.
await fs.rename(part, target);
console.log(`Saved ${bytes} bytes to ${target}`);

// Keep only the most recent backups. Names sort chronologically.
const backups = [];
for await (const entry of fs.readDir(dir)) {
	if (!entry.isDir && entry.name.endsWith(".bak")) {
		backups.push(entry.name);
	}
}
backups.sort();

const stale = backups.slice(0, Math.max(0, backups.length - KEEP));
for (const name of stale) {
	await fs.remove(`${dir}/${name}`);
}

console.log(`Pruned ${stale.length} old backup(s)`);

How It Works#

  • fs.writeFile accepts a ReadableStream. Passing res.body writes the response to disk as it arrives, so memory stays small no matter how large the download is. fs.writeFile creates the file, so there is nothing to set up first.
  • The download goes to a temporary .part file and is renamed to .bak only after it completes. A rename is a single transaction, so the final name never points at a half-written file. If the stream fails, the catch removes the partial file and rethrows.
  • Buffering the whole body first, with await res.arrayBuffer() or by reading the file into a string, would hold the entire payload at once and can exceed the run’s memory limit on a large file.
  • fs.readDir yields directory entries one at a time, so pruning does not load the whole listing.
  • Pruning uses the timestamp in the name: sorted names are chronological, so everything older than the newest KEEP files is removed.

Setup#

Create a secret named BACKUP_URL with the file or API endpoint to download. The job writes to /backups and keeps the seven most recent files. Schedule it daily, for example 0 2 * * *.

An interrupted run can leave a .part file in /backups. It is ignored by the prune, so it is safe, but you can remove it by hand if you want the directory tidy.

Watch the storage quota: each retained backup counts against it, and the quota is separate from the per-run memory limit.