Use S3-Compatible Storage#
S3-compatible object storage is available from many providers, and they all
share the same request-signing scheme: AWS Signature Version 4. The sandbox
does not include an AWS SDK, so this example builds a small helper for basic
object operations that signs requests with Web Crypto, then uses it to upload
and read an object.
The helper supports GET, PUT, HEAD, and DELETE, custom headers, query
parameters for list requests, and both path-style and virtual-hosted URLs.
The Helper#
Save this file as /lib/s3.js. It exports createS3Client, which takes the
bucket configuration and returns an object with a fetch method.
s3.fetch(key, options) signs and sends one request.
- Supports
GET, PUT, HEAD, and DELETE.
- Accepts custom headers and query parameters for list requests.
- Works with path-style and virtual-hosted URLs.
/**
* @fileoverview
* Minimal AWS Signature Version 4 client for S3-compatible storage.
*
* Supports object GET, PUT, HEAD, and DELETE requests, custom headers, and
* query parameters for list requests. Bodies must be strings or Uint8Arrays
* and are hashed in memory, so keep requests small.
*
* Copy this file into your workbench and import it from a job.
*
* @see https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
*/
const encoder = new TextEncoder();
/**
* @typedef {Object} S3ClientConfig
* @property {string|URL} endpoint - S3 endpoint origin, without a path.
* @property {string} bucket - Bucket name.
* @property {string} region - Signing region for the bucket.
* @property {string} accessKeyId - Access key ID.
* @property {string} secretAccessKey - Secret access key.
* @property {boolean} [pathStyle=true] - Use path-style URLs.
* @property {string} [sessionToken] - Token for temporary credentials.
*/
/**
* @typedef {Object} S3RequestOptions
* @property {string} [method="GET"] - HTTP method.
* @property {string|Uint8Array} [body] - Request body.
* @property {Object<string, string|number|boolean>} [headers] - Headers to sign and send.
* @property {Object<string, string|number|boolean>} [query] - Query parameters.
* @property {AbortSignal} [signal] - Cancels the request.
*/
/**
* @typedef {Object} S3Client
* @property {(key: string, options?: S3RequestOptions) => Promise<Response>} fetch
* Sign and send one S3 request.
*/
/**
* Convert bytes to lowercase hex. SigV4 uses hex for hashes and signatures.
*
* @param {ArrayBuffer|Uint8Array} value
* @returns {string}
*/
function toHex(value) {
return [...new Uint8Array(value)]
.map((byte) => byte.toString(16).padStart(2, "0"))
.join("");
}
/**
* Hash a string or byte array with SHA-256.
*
* Large inputs count against the sandbox's Web Crypto budget, which also
* caps the size of a body this client can sign.
*
* @param {string|Uint8Array} value
* @returns {Promise<string>} Lowercase hex digest
*/
async function sha256(value) {
const bytes = typeof value === "string" ? encoder.encode(value) : value;
return toHex(await crypto.subtle.digest("SHA-256", bytes));
}
/**
* Compute HMAC-SHA256. The SigV4 signing key is built by chaining this
* function over the date, region, service, and "aws4_request".
*
* @param {string|Uint8Array} key
* @param {string} value
* @returns {Promise<Uint8Array>}
*/
async function hmac(key, value) {
const rawKey = typeof key === "string" ? encoder.encode(key) : key;
const cryptoKey = await crypto.subtle.importKey(
"raw",
rawKey,
{ name: "HMAC", hash: "SHA-256" },
false,
["sign"],
);
return new Uint8Array(
await crypto.subtle.sign("HMAC", cryptoKey, encoder.encode(value)),
);
}
/**
* Encode one path or query component using SigV4's URI rules.
*
* encodeURIComponent leaves !'()* alone, and SigV4 requires those characters
* to be percent-encoded too.
*
* @param {string} value
* @returns {string}
*/
function encodeRfc3986(value) {
return encodeURIComponent(value).replace(
/[!'()*]/g,
(char) => `%${char.charCodeAt(0).toString(16).toUpperCase()}`,
);
}
// Encode each path segment, keeping the "/" separators intact.
function encodePath(key) {
return key.split("/").map(encodeRfc3986).join("/");
}
/**
* Build the canonical query string: percent-encode each name and value, then
* sort by name and value. SigV4 signs this exact string.
*
* @param {Object<string, string|number|boolean>} query
* @returns {string}
*/
function encodeQuery(query) {
const pairs = [];
for (const [name, value] of Object.entries(query)) {
if (value === undefined || value === null) continue;
pairs.push([encodeRfc3986(name), encodeRfc3986(String(value))]);
}
pairs.sort(([aName, aValue], [bName, bValue]) => {
if (aName < bName) return -1;
if (aName > bName) return 1;
if (aValue < bValue) return -1;
if (aValue > bValue) return 1;
return 0;
});
return pairs.map(([name, value]) => `${name}=${value}`).join("&");
}
/**
* Format a Date as YYYYMMDDTHHMMSSZ.
*
* @param {Date} date
* @returns {string}
*/
function amzDate(date) {
return date.toISOString().replace(/[:-]|\.\d{3}/g, "");
}
/**
* Create a small S3 client that signs requests with AWS Signature Version 4.
*
* @param {S3ClientConfig} config
* @returns {S3Client}
* @throws {TypeError} If a required setting is missing or the endpoint includes a path.
*
* @example
* const s3 = createS3Client({
* endpoint: "https://s3.example.com",
* bucket: "reports",
* region: "us-east-1",
* accessKeyId: "AKIA...",
* secretAccessKey: "...",
* });
*
* const res = await s3.fetch("reports/latest.json", { method: "HEAD" });
*/
export function createS3Client({
endpoint,
bucket,
region,
accessKeyId,
secretAccessKey,
pathStyle = true,
sessionToken,
}) {
if (!endpoint) throw new TypeError("endpoint is required");
if (!bucket) throw new TypeError("bucket is required");
if (!region) throw new TypeError("region is required");
if (!accessKeyId) throw new TypeError("accessKeyId is required");
if (!secretAccessKey) throw new TypeError("secretAccessKey is required");
const base = new URL(endpoint);
if (base.pathname !== "/" && base.pathname !== "") {
throw new TypeError("endpoint must not include a path");
}
return {
/**
* Sign and send one S3 request.
*
* @param {string} key - Object key, or "" for a bucket-level request.
* @param {S3RequestOptions} [options]
* @returns {Promise<Response>} Raw response; check `ok` before reading the body.
* @throws {TypeError} If the body is not a string or Uint8Array.
*/
async fetch(key, options = {}) {
const method = (options.method ?? "GET").toUpperCase();
const body = options.body;
if (
body !== undefined &&
typeof body !== "string" &&
!(body instanceof Uint8Array)
) {
throw new TypeError("body must be a string or Uint8Array");
}
const query = encodeQuery(options.query ?? {});
const search = query ? `?${query}` : "";
const encodedKey = key ? encodePath(key) : "";
// Path-style: https://endpoint/bucket/key
// Virtual-hosted: https://bucket.endpoint/key
const root = pathStyle
? `${base.origin}/${encodeRfc3986(bucket)}`
: `${base.protocol}//${bucket}.${base.host}`;
const url = new URL(
`${root}${encodedKey ? `/${encodedKey}` : ""}${search}`,
);
const timestamp = amzDate(new Date());
const day = timestamp.slice(0, 8);
// A GET or HEAD has no body, but it still signs the hash of "".
const payloadHash = await sha256(body ?? "");
// Host is part of the signature but is added by fetch, so keep it
// out of the request headers. Authorization is added last and is
// never signed.
const signable = {};
for (const [name, value] of Object.entries(options.headers ?? {})) {
const lower = name.toLowerCase();
if (lower === "host" || lower === "authorization") continue;
if (value === undefined || value === null) continue;
signable[lower] = String(value).trim().replace(/\s+/g, " ");
}
signable.host = url.host;
signable["x-amz-content-sha256"] = payloadHash;
signable["x-amz-date"] = timestamp;
if (sessionToken) {
signable["x-amz-security-token"] = sessionToken;
}
const names = Object.keys(signable).sort();
const canonicalHeaders = `${names
.map((name) => `${name}:${signable[name]}`)
.join("\n")}\n`;
const signedHeaders = names.join(";");
// Canonical request: method, path, query, headers, signed header
// names, and payload hash. canonicalHeaders already ends with a
// newline, so the join below leaves the required blank line.
const canonicalRequest = [
method,
url.pathname,
query,
canonicalHeaders,
signedHeaders,
payloadHash,
].join("\n");
const scope = `${day}/${region}/s3/aws4_request`;
const stringToSign = [
"AWS4-HMAC-SHA256",
timestamp,
scope,
await sha256(canonicalRequest),
].join("\n");
// Derive the signing key: secret -> date -> region -> service ->
// "aws4_request". Only the final signature is sent.
let signingKey = await hmac(`AWS4${secretAccessKey}`, day);
signingKey = await hmac(signingKey, region);
signingKey = await hmac(signingKey, "s3");
signingKey = await hmac(signingKey, "aws4_request");
const signature = toHex(await hmac(signingKey, stringToSign));
const headers = {};
for (const name of names) {
if (name !== "host") {
headers[name] = signable[name];
}
}
headers.authorization =
`AWS4-HMAC-SHA256 Credential=${accessKeyId}/${scope}, ` +
`SignedHeaders=${signedHeaders}, Signature=${signature}`;
const init = { method, headers };
if (body !== undefined) {
init.body = body;
}
if (options.signal !== undefined) {
init.signal = options.signal;
}
return fetch(url.toString(), init);
},
};
}
Using the Helper#
Save this as /upload-to-s3.js. It reads the settings from secrets, creates a
client, uploads a JSON object, and reads it back.
// upload-to-s3.js
import env from "env";
import { createS3Client } from "./lib/s3.js";
const sessionToken = env.names().includes("S3_SESSION_TOKEN")
? env.get("S3_SESSION_TOKEN")
: undefined;
const s3 = createS3Client({
endpoint: env.get("S3_ENDPOINT"),
bucket: env.get("S3_BUCKET"),
region: env.get("S3_REGION"),
accessKeyId: env.get("S3_ACCESS_KEY_ID"),
secretAccessKey: env.get("S3_SECRET_ACCESS_KEY"),
pathStyle: env.get("S3_PATH_STYLE") !== "false",
sessionToken,
});
const report = JSON.stringify({
generatedAt: new Date().toISOString(),
source: "geartrain",
});
const upload = await s3.fetch("reports/latest.json", {
method: "PUT",
body: report,
headers: { "content-type": "application/json" },
});
if (!upload.ok) {
throw new Error(`Upload failed: ${upload.status} ${await upload.text()}`);
}
const download = await s3.fetch("reports/latest.json");
if (!download.ok) {
throw new Error(`Download failed: ${download.status} ${await download.text()}`);
}
const stored = await download.json();
console.log(JSON.stringify(stored, null, 2));
Other operations use the same client:
// Check an object without downloading it.
const head = await s3.fetch("reports/latest.json", { method: "HEAD" });
// List objects under a prefix.
const list = await s3.fetch("", {
query: { "list-type": "2", prefix: "reports/" },
});
// Delete an object.
await s3.fetch("reports/latest.json", { method: "DELETE" });
How It Works#
- SigV4 signs the method, path, query string, signed headers, and a SHA-256
hash of the body. The service recomputes the same values and compares the
signature.
- The helper adds
x-amz-content-sha256 and x-amz-date, signs every header
you pass, and derives host from the URL. fetch adds the Host header
itself.
- The signing key is an HMAC chain over the secret, date, region,
s3, and
aws4_request. Only the derived signature is sent.
pathStyle controls the URL shape. Path-style uses
/<bucket>/<key>; virtual-hosted uses <bucket>.<endpoint>/<key>.
- Query names and values are percent-encoded and sorted before signing, which
is required for list requests.
- The helper returns a normal
Response. Check res.ok before reading the
body, as with fetch.
Limits#
- This is a small client for basic object operations, not a full S3 SDK.
- Requests are single-part. Multipart uploads, streaming bodies, automatic
retries, and redirects are not implemented.
- Bodies are hashed in memory, so keep objects small. Hashing and signing
also count against the run’s Web Crypto budget.
- The plan’s memory, compute, and run limits still apply. See
Limits.
- Third-party modules are not available yet, but support is planned. That
would allow a full-featured S3 client to run in your scripts. Until then,
call the service’s HTTP API directly, as this example does. See
The Sandbox.
Setup#
-
Create a bucket and an access key with PutObject and GetObject
permissions. Add ListBucket and DeleteObject if you use the list or
delete snippets. Scope the key to the bucket.
-
Add the secrets:
| Secret |
Value |
S3_ENDPOINT |
Your provider’s S3 endpoint, without a path |
S3_BUCKET |
The bucket name |
S3_REGION |
Your provider’s region |
S3_PATH_STYLE |
true for path-style URLs, false for virtual-hosted |
S3_ACCESS_KEY_ID |
The access key ID |
S3_SECRET_ACCESS_KEY |
The secret access key |
S3_SESSION_TOKEN |
Optional, for temporary credentials |
-
Save the helper at /lib/s3.js and the script at /upload-to-s3.js.
-
Open Jobs, click Create Job, select upload-to-s3.js as the
script, name the job, and choose a schedule. Click Create Job.
-
Click Run now on the job, then
check reports/latest.json in the bucket.
A script does not run on its own. The job points at it and supplies the
schedule; Run now starts that job immediately.
For virtual-hosted URLs, use a bucket name without dots so the TLS
certificate matches the host.