Scrape a Page With a Service#

Start with a scraping service when a job needs to read a website. The service handles the fetch, and many services also render JavaScript, manage proxies, and return extracted data, so the script stays small and does not need to parse HTML.

There are many scraping APIs to choose from, and the pattern is similar across them: send a URL and the fields you want, then read the structured response. This example uses ScrapingBee, which supports CSS extraction rules; another service with the same feature works the same way once you change the endpoint, credentials, and parameter names.

The target is Books to Scrape, a public sandbox built for scraping practice. The API does the fetching and the extraction; the script sends selectors and stores the result.

The Script#

// scrape-books.js
//
// ScrapingBee fetches the page and applies these rules, so this script never
// downloads or parses HTML. When the service can extract data, let it: the
// job stays small, and a layout change means updating selectors instead of
// rewriting parsing code.
import env from "env";
import fs from "fs";

// Read the key from a saved secret instead of hard-coding it.
const apiKey = env.get("SCRAPINGBEE_API_KEY");

// A public sandbox built for scraping practice.
const target =
	"https://books.toscrape.com/catalogue/category/books/mystery_3/index.html";

// Each key becomes a field in the response. This rule walks every product
// article and returns a list of books. The title comes from the link's title
// attribute, which holds the full name even when the visible text is cut
// short; the price comes from the visible text.
const rules = {
	books: {
		selector: "article.product_pod",
		type: "list",
		output: {
			title: { selector: "h3 a", output: "@title" },
			price: { selector: ".price_color", output: "text" },
		},
	},
};

const endpoint = new URL("https://app.scrapingbee.com/api/v1");
endpoint.searchParams.set("url", target);

// The page is static HTML, so skip the headless browser. This costs 1 credit
// per request instead of 5.
endpoint.searchParams.set("render_js", "false");

// Send the rules as JSON. The response will already be structured.
endpoint.searchParams.set("extract_rules", JSON.stringify(rules));

const res = await fetch(endpoint.toString(), {
	headers: { authorization: `Bearer ${apiKey}` },
});

if (!res.ok) {
	throw new Error(`ScrapingBee error: ${res.status} ${await res.text()}`);
}

// The response is already { books: [...] }, shaped by the rules above.
const { books = [] } = await res.json();

if (books.length === 0) {
	throw new Error("No books extracted; the page markup may have changed");
}

// The script's only remaining job is to store the result.
await fs.writeTextFile("/books.json", JSON.stringify(books, null, 2));
console.log(`Saved ${books.length} books to /books.json`);

How It Works#

  • The request sends the target URL and the extraction rules. ScrapingBee returns JSON, so the script never downloads or parses HTML.
  • books is the response key. type: "list" returns every product, and each key in output becomes a field. The title comes from the link’s title attribute; the price is the visible text.
  • render_js is off because the page is static HTML. That costs 1 credit per request instead of 5.
  • The key comes from a saved secret, never from the script.
  • An empty books array means the selectors no longer match; the script fails instead of saving an empty file.
  • Prices arrive as text. Use Number(book.price.replace(/[^0-9.]/g, "")) if you need to calculate with them.
  • The page lists 20 of 32 books. Each additional page is another request.

Setup#

  1. Create a ScrapingBee account and copy the API key.
  2. Open Secrets, create SCRAPINGBEE_API_KEY, and paste the key.
  3. In the workbench, use Add File to create scrape-books.js, then paste the script.
  4. Open Jobs, click Create Job, select scrape-books.js as the script, name the job, and choose a schedule. Click Create Job.
  5. Click Run now on the job. The result is saved to /books.json.
  6. Change target when you are ready for your own page, and use a page you have permission to scrape; check its robots.txt before you put the job on a schedule.

A script does not run on its own. The job points at it and supplies the schedule; Run now starts that job immediately.