A scraper that tells you when a page changes

Why this one

This is the honest starting point for scraping: no browser, no framework, about sixty lines. A surprising share of "we need to monitor our competitors" or "tell me when this comes back in stock" is exactly this, and people routinely build it with a full browser automation stack that's a hundred times heavier.

It also teaches the thing that separates a script from a tool: state. A scraper that prints the current price is a toy. One that remembers the last price and tells you only when it changed is something you'll actually keep running.

Before you start

You need Python 3.10 or newer. Nothing else — no accounts, no API keys, no costs.

⚠ Scrape considerately. Read the site's robots.txt, keep the delay between requests, and don't point this at a login-protected page. The tools guide has a fuller section on where the lines are.

Step 1 — Install

pip install requests beautifulsoup4 lxml

Three packages: requests fetches, beautifulsoup4 parses, lxml is the fast parsing engine BeautifulSoup will use. SQLite is already in Python.

Step 2 — Find your selectors

For each page you want to watch, open it, right-click the value you care about and choose Inspect. You're looking for a CSS selector that reliably targets it.

Good selectors are anchored to something meaningful: .price, [data-testid="stock-status"], #main h1. Bad selectors are anchored to generated class names like .css-1x7ab3k — those are build artefacts and will change without warning.

Test a selector in the browser console before trusting it:

document.querySelector('.price').innerText

Step 3 — The watcher

Save as watch.py. This is the complete file — it's the exact code that produced the output further down.

"""Watch pages for changes. Tells you only when something moved."""
import sqlite3, hashlib, time, sys
import requests
from bs4 import BeautifulSoup

# --- what to watch ---------------------------------------------------
WATCHING = [
    {"name": "Widget Pro price", "url": "https://example.com/widget", "select": ".price"},
    {"name": "Widget Pro stock", "url": "https://example.com/widget", "select": ".stock"},
]
# Identify yourself. A real contact address means a site owner can reach
# you instead of silently blocking you.
HEADERS = {"User-Agent": "yourname-watcher/1.0 (+https://yoursite.com)"}
POLITE_DELAY = 2          # seconds between requests
# ---------------------------------------------------------------------


def db():
    c = sqlite3.connect("watcher.db")
    c.execute("""CREATE TABLE IF NOT EXISTS seen(
        name TEXT PRIMARY KEY, fingerprint TEXT, value TEXT, checked_at TEXT)""")
    return c


def grab(watch):
    r = requests.get(watch["url"], headers=HEADERS, timeout=20)
    r.raise_for_status()
    # NOTE: r.content (bytes), not r.text. See "when it goes wrong".
    soup = BeautifulSoup(r.content, "lxml")
    el = soup.select_one(watch["select"])
    if el is None:
        raise LookupError(f"Selector {watch['select']!r} matched nothing")
    # Collapse whitespace so "  £10.00\n " and "£10.00" aren't a change.
    return " ".join(el.get_text().split())


def check(conn, watch):
    value = grab(watch)
    fingerprint = hashlib.sha256(value.encode()).hexdigest()[:16]
    row = conn.execute(
        "SELECT fingerprint, value FROM seen WHERE name=?", (watch["name"],)).fetchone()
    conn.execute(
        """INSERT INTO seen(name,fingerprint,value,checked_at)
           VALUES(?,?,?,datetime('now'))
           ON CONFLICT(name) DO UPDATE SET fingerprint=excluded.fingerprint,
           value=excluded.value, checked_at=excluded.checked_at""",
        (watch["name"], fingerprint, value))
    conn.commit()
    if row is None:
        return ("first", value, None)
    if row[0] != fingerprint:
        return ("changed", value, row[1])
    return ("same", value, None)


def main():
    conn = db()
    for i, w in enumerate(WATCHING):
        if i:
            time.sleep(POLITE_DELAY)
        try:
            state, now, before = check(conn, w)
        except Exception as e:
            # One broken page must never stop the others.
            print(f"[error] {w['name']}: {e}", file=sys.stderr)
            continue
        if state == "first":
            print(f"[baseline] {w['name']}: {now}")
        elif state == "changed":
            print(f"[CHANGED] {w['name']}: {before} -> {now}")
        else:
            print(f"[same] {w['name']}: {now}")


if __name__ == "__main__":
    main()

Step 4 — Run it

First run records a baseline. Later runs compare against it. Here's the real output from three consecutive runs, with the price edited between the second and third:

$ python watch.py            # first run
[baseline] Widget Pro price: £1,299.00
[baseline] Widget Pro stock: In stock

$ python watch.py            # nothing moved
[same] Widget Pro price: £1,299.00
[same] Widget Pro stock: In stock

$ python watch.py            # after the price changed
[CHANGED] Widget Pro price: £1,299.00 -> £1,149.00
[same] Widget Pro stock: In stock

And a deliberately broken selector, to show it degrades sensibly rather than crashing:

[error] nope: Selector '.does-not-exist' matched nothing

Step 5 — Tell yourself about it

Printing to a terminal nobody's watching isn't much use. Add a notifier and call it in the changed branch:

import requests

TELEGRAM_TOKEN = "..."     # from @BotFather
TELEGRAM_CHAT  = "..."     # your chat id

def notify(text):
    requests.post(
        f"https://api.telegram.org/bot{TELEGRAM_TOKEN}/sendMessage",
        json={"chat_id": TELEGRAM_CHAT, "text": text},
        timeout=20,
    )

        elif state == "changed":
            message = f"{w['name']}\n{before} → {now}\n{w['url']}"
            print("[CHANGED]", message)
            notify(message)

Keeping the fetching, the comparing and the notifying as separate functions is deliberate — swapping Telegram for Slack or email touches one function and nothing else.

Step 6 — Put it on a schedule

On a machine that's reliably on, crontab -e:

0 * * * * cd /path/to/project && /usr/bin/python3 watch.py >> watch.log 2>&1

If your laptop isn't always on, a scheduled GitHub Action is free and always running. One wrinkle: the Action gets a fresh filesystem each run, so watcher.db won't survive. Either commit it back to the repository at the end of the run, or use the Actions cache — otherwise every run looks like a first run and you'll be told everything changed, every time.

When it goes wrong

Prices appear as £1,299.00. This is the classic one, and it bit us while testing this project. When a server doesn't declare its character set, requests assumes ISO-8859-1 and r.text gives you mojibake. Passing r.content — raw bytes — lets the parser read the page's own declared encoding. Here's the actual difference:

declared encoding: ISO-8859-1 | apparent: utf-8
r.text    -> £1,149.00
r.content -> £1,149.00

It reports a change every single run. Something in your selected element changes on every load — a timestamp, a session id, a rotating advert. Narrow the selector, or strip the volatile part before fingerprinting.

Selector matched nothing, but it works in the browser. The content is rendered by JavaScript, so it isn't in the HTML requests received. Confirm with curl: if the value isn't in the raw response, you need the Playwright approach — or, better, the JSON endpoint trick that build sheet opens with.

403 or 429 responses. 403 usually means the default Python User-Agent got you blocked — set a real one. 429 means you're going too fast; raise POLITE_DELAY.

It worked for three weeks then stopped. The site redesigned. This is normal and permanent; scrapers are maintenance. Saving the raw HTML alongside the value turns the fix into five minutes.

What this costs

Nothing. No API keys, no model calls, no server if you use GitHub Actions. The only cost is the occasional twenty minutes fixing a selector after a site redesign.

Make it yours

Watch numbers, not strings. Parse the price to a float and only alert when it drops, or moves more than 5%. Far fewer, far more useful messages.

Keep history. Write every observation to a second table instead of overwriting. You get a price history chart for free, which is often worth more than the alerts.

Store the raw HTML. Add a column for the full response. Future-you fixing a broken selector will be grateful.

Add judgment. For pages where "did it change" is too crude — a policy document, a competitor's homepage — send the before and after to a model and ask whether the change is meaningful. That's the bridge from this project to the agents track.