Scraping a site that builds itself with JavaScript

The problem, demonstrated

You point the change watcher at a job board and get nothing. The data is plainly there in your browser. Here's what requests actually received:

job elements found: 0
page text: JobsLoading…

The server sent a nearly empty page. The content arrives afterwards, when the page's JavaScript fetches it and writes it into the DOM. requests doesn't run JavaScript, so it sees the "Loading…" and stops.

Do this before you reach for a browser

That JavaScript is fetching the data from somewhere. Usually a JSON endpoint on the same site. If you call that directly you skip the browser entirely.

Open developer tools, go to the Network tab, filter to Fetch/XHR, and reload. Look through the responses for the data you want. When you find it, the URL is your scraper.

Here's the payoff on the test page used throughout this build sheet. Through Playwright, we can log which JSON the page requested:

 - Backend Engineer | Remote
 - Data Analyst | Ottawa
JSON endpoints the page called: ['https://example.com/jobs.json']

And that endpoint hit directly, no browser at all:

import requests
data = requests.get("https://example.com/jobs.json", timeout=20).json()
for j in data["jobs"]:
    print(j["title"], j["location"], j["salary"])

 - Backend Engineer | Remote | $140k
 - Data Analyst | Ottawa | $95k

Note the salary. The JSON contained a field the rendered page never displayed. That's common — the API returns the full record and the interface shows a subset — and it's a good argument for always looking.

Sometimes there's no usable endpoint — it's authenticated, signed, or the HTML really is assembled client-side from fragments. Then you need the browser. That's the rest of this page.

Step 1 — Install

pip install playwright
playwright install chromium

The second command downloads a browser — a few hundred megabytes. That weight is the cost you're accepting, and the reason to check for the API first.

Step 2 — The scraper

Save as scrape.py. Complete file.

"""Scrape a JavaScript-rendered page with Playwright, into SQLite."""
import sqlite3, time
from playwright.sync_api import sync_playwright, TimeoutError as PWTimeout

URL = "https://example.com/jobs"
ROW = ".job"              # each result
FIELDS = {"title": ".t", "location": ".l"}   # name -> selector within a row
UA = "yourname-scraper/1.0 (+https://yoursite.com)"


def db():
    c = sqlite3.connect("scraped.db")
    c.execute("""CREATE TABLE IF NOT EXISTS jobs(
        title TEXT, location TEXT, seen_at TEXT,
        UNIQUE(title, location))""")
    return c


def scrape():
    rows, endpoints = [], []
    with sync_playwright() as p:
        browser = p.chromium.launch()          # headless by default
        page = browser.new_page(user_agent=UA)

        # Log any JSON the page fetches — this is how you find the API
        # you could have called instead of running all this.
        page.on("response", lambda r: endpoints.append(r.url)
                if "json" in r.headers.get("content-type", "") else None)

        page.goto(URL, wait_until="networkidle", timeout=30000)

        # Wait for real content, not a fixed sleep. This is the single
        # biggest practical advantage Playwright has over older tools.
        try:
            page.wait_for_selector(ROW, timeout=15000)
        except PWTimeout:
            page.screenshot(path="failed.png", full_page=True)
            raise SystemExit(
                f"No {ROW!r} appeared in 15s. Saved failed.png — look at what "
                f"the page actually showed (a cookie wall? a bot check?)")

        for el in page.query_selector_all(ROW):
            row = {}
            for name, sel in FIELDS.items():
                found = el.query_selector(sel)
                row[name] = found.inner_text().strip() if found else None
            rows.append(row)

        browser.close()
    return rows, endpoints


def save(conn, rows):
    added = 0
    for r in rows:
        cur = conn.execute(
            """INSERT OR IGNORE INTO jobs(title, location, seen_at)
               VALUES(?,?,datetime('now'))""",
            (r["title"], r["location"]))
        added += cur.rowcount
    conn.commit()
    return added


if __name__ == "__main__":
    rows, endpoints = scrape()
    print(f"Found {len(rows)} rows")
    for r in rows:
        print("  -", r["title"], "|", r["location"])
    added = save(db(), rows)
    print(f"{added} new, {len(rows) - added} already seen")
    if endpoints:
        print("\nJSON the page fetched — check whether you can use it directly:")
        for u in dict.fromkeys(endpoints):
            print("  ", u)

Step 3 — Run it

$ python scrape.py
Found 2 rows
  - Backend Engineer | Remote
  - Data Analyst | Ottawa
2 new, 0 already seen

JSON the page fetched — check whether you can use it directly:
   https://example.com/jobs.json

Run it twice and the second run reports 0 new, 2 already seen — the UNIQUE constraint plus INSERT OR IGNORE does deduplication for you, with no extra code.

Three things that make browser scraping bearable

Wait for elements, never for time. page.wait_for_selector(...) continues the instant the thing appears. time.sleep(5) is both slower on a fast day and broken on a slow one. Playwright's auto-waiting also applies to clicks and typing, which removes most of the flakiness people associate with browser automation.

Screenshot on failure. The code above saves failed.png when the selector never appears. Nine times in ten it immediately shows you the answer: a cookie banner covering the page, a region block, a login wall, or a bot check. Debugging blind here wastes hours.

Watch it work while developing. Launch with p.chromium.launch(headless=False, slow_mo=400) and you see exactly what the script sees. Switch back to headless once it works.

When it goes wrong

Works headed, fails headless. A real difference — some sites detect headless mode, and viewport size can change what renders. Try setting a viewport and a normal User-Agent first; if it's a deliberate bot check, that's the site telling you something, and worth respecting rather than escalating.

A cookie banner blocks everything. Dismiss it before scraping: page.click("text=Accept", timeout=3000) inside a try/except so it doesn't break when the banner isn't shown.

Content loads as you scroll. Infinite scroll needs a loop: scroll to the bottom, wait for the row count to rise, repeat until it stops — with a maximum, or you'll scroll forever.

previous = 0
for _ in range(20):
    page.mouse.wheel(0, 20000)
    page.wait_for_timeout(1200)
    count = len(page.query_selector_all(ROW))
    if count == previous:
        break
    previous = count

It's unbearably slow. Expect a few seconds per page — that's the browser. If you need many pages, reuse one browser across all of them rather than launching per page, and run several contexts concurrently. And re-read the API section at the top.

Timeouts on a slow site. wait_until="networkidle" can hang on pages that poll continuously. Try "domcontentloaded" plus an explicit wait_for_selector instead.

What this costs

Free to run, but not free in resources: each browser instance wants a few hundred megabytes of RAM and a few seconds per page. On a free CI runner that's fine for tens of pages and painful for thousands — which is the point at which you either find the API or move to a crawling framework.

Make it yours

Go find the API properly. Take the endpoints the script logged, open one in your browser, and see whether it works without the session. If it does, delete the browser code — your scraper just got a hundred times faster.

Handle pagination. Loop over ?page=1..n, or click the next button until it disappears. Save after each page so a crash doesn't cost you the run.

Log in once, reuse the session. context.storage_state(path="auth.json") saves cookies; load it next run and skip the login. Only for sites whose terms permit it.

Alert on new rows. You already know which rows are new — wire the notifier from the change watcher and you have a job alert that beats the site's own.