Crawling an entire site without getting blocked

Why this one

The single-page scraper is a solved problem. What breaks at scale is everything around the fetch: you need to not hit the same page twice, not hammer the server, not lose three hours of progress when your laptop sleeps, and not end up with a dataset where half the rows are the same page under a different URL.

Scrapy handles most of that out of the box, which is exactly why it's the right tool — and exactly why its failures are quiet. Every bug in this sheet produced a crawl that finished, reported success, and returned the wrong data.

Before you start

Python 3.10+, and a site you're allowed to crawl. Built and tested with Scrapy 2.19 against a 41-page test site built to trip every failure below, and against this site.

⚠ Crawl sites you own or have permission to crawl. Obeying robots.txt is the floor, not the ceiling — read the site's terms, and put a real contact address in the user agent so an admin can reach you instead of blocking you.

Step 1 — Install

pip install scrapy

That's the only dependency. SQLite ships with Python.

Step 2 — The crawler

Save as crawl.py. This is the complete file.

"""
A polite whole-site crawler: robots.txt, self-throttling, resumable, SQLite out.

    python crawl.py https://example.com            # crawl
    python crawl.py https://example.com            # run again: resumes, doesn't refetch
"""
import hashlib, re, sqlite3, sys
from datetime import datetime, timezone
from urllib.parse import urlparse, urlencode, parse_qsl, urlunparse

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.linkextractors import LinkExtractor

TRACKING = re.compile(r"^(utm_|fbclid$|gclid$|mc_|ref$|ref_)")


def clean_url(url):
    """One page, one URL: drop fragments and tracking params, sort the rest."""
    p = urlparse(url)
    q = sorted((k, v) for k, v in parse_qsl(p.query) if not TRACKING.match(k))
    return urlunparse((p.scheme, p.netloc.lower(), p.path or "/", "", urlencode(q), ""))


class SiteSpider(scrapy.Spider):
    name = "site"

    def __init__(self, start, **kw):
        super().__init__(**kw)
        self.start_urls = [start]
        # Filter domains in ONE place. Scrapy has two domain filters that disagree
        # about ports: allowed_domains compares hostnames ("localhost"), but
        # LinkExtractor(allow_domains=...) compares host-plus-port ("localhost:8000").
        # Give both the same value and one of them silently drops every link.
        self.allowed_domains = [urlparse(start).hostname]
        self.links = LinkExtractor(process_value=clean_url)

    def parse(self, response):
        if "text/html" not in response.headers.get("Content-Type", b"").decode():
            return
        text = " ".join(" ".join(response.css("body *::text").getall()).split())
        yield {
            "url": clean_url(response.url),
            "status": response.status,
            "title": (response.css("title::text").get() or "").strip(),
            "h1": (response.css("h1::text").get() or "").strip(),
            "description": response.css('meta[name="description"]::attr(content)').get() or "",
            "canonical": response.css('link[rel="canonical"]::attr(href)').get() or "",
            "words": len(text.split()),
            # same text, different URL (?page=2, ?sort=, session ids) -> same hash
            "hash": hashlib.sha1(text.encode()).hexdigest()[:16],
            "depth": response.meta.get("depth", 0),
            "fetched": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        }
        for link in self.links.extract_links(response):
            yield response.follow(link.url, callback=self.parse)


class SQLitePipeline:
    """UPSERT on url: a re-run or resume updates rows instead of duplicating them."""

    def open_spider(self):
        self.db = sqlite3.connect("pages.db")
        self.db.execute("""CREATE TABLE IF NOT EXISTS pages (
            url TEXT PRIMARY KEY, status INT, title TEXT, h1 TEXT, description TEXT,
            canonical TEXT, words INT, hash TEXT, depth INT, fetched TEXT)""")

    def process_item(self, item):
        self.db.execute("""INSERT INTO pages VALUES
            (:url,:status,:title,:h1,:description,:canonical,:words,:hash,:depth,:fetched)
            ON CONFLICT(url) DO UPDATE SET status=excluded.status, title=excluded.title,
            h1=excluded.h1, description=excluded.description, canonical=excluded.canonical,
            words=excluded.words, hash=excluded.hash,
            depth=MIN(pages.depth, excluded.depth),   -- the homepage gets fetched twice
            fetched=excluded.fetched""", item)
        self.db.commit()
        return item

    def close_spider(self):
        self.db.close()


if __name__ == "__main__":
    start = sys.argv[1]
    process = CrawlerProcess(settings={
        "USER_AGENT": "polite-crawler/1.0 (+mailto:you@example.com)",
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,          # slows down when the server does
        "AUTOTHROTTLE_START_DELAY": 1.0,
        "AUTOTHROTTLE_TARGET_CONCURRENCY": 1.0,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
        "DOWNLOAD_DELAY": 0.5,
        "RETRY_TIMES": 3,
        "RETRY_HTTP_CODES": [429, 500, 502, 503, 504],
        "JOBDIR": "crawl-state",               # this line is what makes it resumable
        "DEPTH_LIMIT": 10,
        "CLOSESPIDER_PAGECOUNT": int(sys.argv[2]) if len(sys.argv) > 2 else 0,
        "ITEM_PIPELINES": {"__main__.SQLitePipeline": 100},
        "LOG_LEVEL": "INFO",
    })
    process.crawl(SiteSpider, start=start)
    process.start()

Four settings do most of the work. ROBOTSTXT_OBEY fetches robots.txt first and skips anything disallowed. AUTOTHROTTLE_ENABLED measures how fast the server answers and slows down when it struggles. JOBDIR writes the queue and the list of seen URLs to disk, which is what makes it resumable. RETRY_HTTP_CODES includes 429, so being told to slow down is treated as a retry, not an error.

clean_url is what keeps the dataset clean at the source: the same page linked as /post, /post#comments and /post?utm_source=newsletter becomes one URL before Scrapy ever decides whether it has seen it.

Step 3 — Run it

$ python crawl.py https://letsbuildabot.com/
...
 'dupefilter/filtered': 613,
 'finish_reason': 'finished',
 'item_scraped_count': 35,
 'offsite/filtered': 36,

real    0m24.5s

Read the stats, not the page count. dupefilter/filtered: 613 is Scrapy declining to fetch pages it had already queued — on a real site almost every page links to the same navigation. offsite/filtered: 36 is links to other domains being ignored. Thirty-five pages in 24 seconds is the throttle doing its job; against the local test site, 82 requests took 49 seconds and never exceeded three in any one second.

Step 4 — Stop it and start it again

Test resuming before you need it. The optional second argument stops the crawl after roughly that many pages:

$ python crawl.py https://example.com/ 15      # stop after ~15 pages
 'finish_reason': 'closespider_pagecount',
 'item_scraped_count': 29,

$ python crawl.py https://example.com/         # run it again
INFO: Resuming crawl (27 requests scheduled)
 'finish_reason': 'finished',
 'item_scraped_count': 53,

It picked up the 27 requests still in the queue and carried on. Across both runs, no page was fetched twice except the homepage — more on that below. To start from scratch, delete the crawl-state folder.

Step 5 — Get a clean dataset out

Save as export.py:

import csv, sqlite3

db = sqlite3.connect("pages.db")
# One row per distinct page: when several URLs share a content hash,
# keep the shortest (usually the one without ?sort= or ?page=).
rows = db.execute("""
    SELECT url, title, h1, description, canonical, words, depth FROM pages
    WHERE url IN (
        SELECT url FROM (
            SELECT url, ROW_NUMBER() OVER (
                PARTITION BY hash ORDER BY length(url), url) AS rn
            FROM pages)
        WHERE rn = 1)
    ORDER BY url""").fetchall()

with open("pages.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["url", "title", "h1", "description", "canonical", "words", "depth"])
    w.writerows(rows)

total = db.execute("SELECT count(*) FROM pages").fetchone()[0]
print(f"{len(rows)} distinct pages -> pages.csv  ({total - len(rows)} duplicate URLs dropped)")

$ python export.py
29 distinct pages -> pages.csv  (5 duplicate URLs dropped)

The five dropped rows are real: this site's projects page has track filters linked as /projects/?track=agents and so on. The filtering happens in the browser, so every one of those URLs serves identical HTML. Without the content hash you would count them as five extra pages.

Step 6 — Check it against the sitemap

A crawler can only find what's linked. Comparing against the sitemap in both directions is the cheapest completeness check there is. Save as reconcile.py:

import re, sqlite3, sys
import requests

site = sys.argv[1].rstrip("/")
db = sqlite3.connect("pages.db")
crawled = {u for (u,) in db.execute("SELECT url FROM pages")}
xml = requests.get(site + "/sitemap.xml", timeout=30).text
listed = set(re.findall(r"<loc>\s*([^<\s]+)\s*</loc>", xml))

print(f"crawled {len(crawled)}, sitemap lists {len(listed)}\n")
print("In the sitemap, never reached by following links:")
for u in sorted(listed - crawled) or ["  (none)"]:
    print("  ", u)
print("\nCrawled, but not in the sitemap:")
for u in sorted(crawled - listed) or ["  (none)"]:
    print("  ", u)

$ python reconcile.py https://letsbuildabot.com
crawled 34, sitemap lists 29

In the sitemap, never reached by following links:
   (none)

Crawled, but not in the sitemap:
   https://letsbuildabot.com/projects/?track=agents
   https://letsbuildabot.com/projects/?track=ai-in-things
   https://letsbuildabot.com/projects/?track=programming
   https://letsbuildabot.com/projects/?track=robots
   https://letsbuildabot.com/projects/?track=scrapers

Nothing in the sitemap was unreachable, so the crawl is complete. The five extras are the filter URLs from Step 5 — correctly absent from the sitemap, and correctly flagged as duplicates by the hash. When the first list isn't empty, those pages are orphans: nothing links to them, so a crawler — and possibly a search engine — never finds them.

When it goes wrong

It crawls the homepage and stops, reporting success. This one cost the most time, twice, for two different reasons — both caused by a port number. Scrapy has two domain filters that disagree about ports. The spider's allowed_domains compares hostnames (localhost); LinkExtractor(allow_domains=...) compares host-plus-port (localhost:8000). Give both the same value and one of them silently discards every link. On a real site with no port in the URL they agree, which is why this only bites when you test locally. The fix, used above: filter domains in exactly one place. scheduler/enqueued: 1 in the stats is the tell.

Twice as many rows as the site has pages. Query parameters that don't change the content — ?page=2, ?sort=newest, filter views, session IDs — make the same page look new. The test site had 41 pages and produced 81 rows. clean_url strips tracking parameters, but it can't know which of the others matter. The content hash can: same text, same hash.

The homepage is fetched twice, and its depth is wrong. Scrapy exempts start URLs from duplicate filtering, so the first page linking back to / fetches it again — and a naive upsert records the second visit's depth. That's why the pipeline keeps MIN(depth). One extra request is harmless; a homepage at depth 1 quietly breaks any analysis of site structure.

You asked for 15 pages and got 29. CLOSESPIDER_PAGECOUNT stops *scheduling* at the limit; requests already in flight still complete. It's a soft stop, which is fine for testing resume and wrong for anything that needs an exact cap.

Pages that exist but never show up. Links built by JavaScript are invisible to Scrapy, which reads the HTML it was sent. The same thing bit a crawler used to recover this very site from its live deployment: its data files are loaded with fetch(base + "data/" + f), no pattern-matcher can see a URL assembled at runtime, and it reported zero unresolved references while missing all eight. The sitemap check catches this. For sites that build themselves in the browser, see scraping a site that builds itself with JavaScript.

“Zero errors” is not “complete.” A crawler's error count only covers what it attempted. Every failure on this page produced a run with no errors. Check the output against something independent — the sitemap, a known page count — before trusting it.

429s and 503s. The retry settings back off on these automatically. If they keep coming, lower AUTOTHROTTLE_TARGET_CONCURRENCY to 0.5 and raise DOWNLOAD_DELAY. A site that keeps rejecting you after that is telling you something — ask for access or an export instead.

Deprecation warnings about pipeline methods. Scrapy 2.19 no longer passes spider to open_spider, process_item and close_spider. Older tutorials include it; the version above doesn't.

What this costs

Nothing to build or run. No API keys, no model calls. Time is the real cost: at a polite one to two requests a second, ten thousand pages is roughly two hours. That's the right speed — if you need it faster, you need permission, not concurrency.

Make it yours

Respect the canonical tag. You're already storing it. Group by canonical as well as by hash and you'll catch duplicates whose text differs slightly — a date, a view counter.

Crawl on a schedule and diff. Keep each run's pages.db, and compare: new pages, removed pages, pages whose hash changed. That's a change-monitor for a whole site, and pairs naturally with a scraper that tells you when a page changes.

Extract what you actually want. The parse method is where the real work goes. Prices, dates, product specs — add fields to the item and columns to the table, and keep the page-level fields for debugging.

Find your own broken links. Point it at your own site and add a callback for non-200 responses (handle_httpstatus_list). A site-wide link checker is about ten more lines.

Find thin pages. SELECT url, words FROM pages ORDER BY words LIMIT 20 — the pages with the least content are usually the ones search engines ignore.