requests, BeautifulSoup, Selenium, Playwright, Scrapy: which one and when

First, what each one actually is

People compare these as if choosing one rules out the others. In fact they occupy three different layers.

requests — fetches

Makes an HTTP request and hands you back bytes. That's the whole job. It does not understand HTML, cannot click anything, and has no idea what JavaScript is. It's the plumbing underneath most Python scraping, and it is extremely fast because it does so little.

BeautifulSoup — parses

Takes HTML and lets you find things in it: soup.select_one(".price"). It does not fetch anything. You hand it the bytes that requests fetched. This pairing — requests to fetch, BeautifulSoup to parse — is the default scraping stack and handles a genuinely large share of real jobs.

The common confusion: "should I use requests or BeautifulSoup?" is like asking whether to use a kettle or a teapot.

Selenium — drives a browser

Launches a real browser and controls it. Originally built for testing, widely used for scraping because it executes JavaScript, holds a login session, and can click. It's mature, it supports every language and browser, and it is slower and heavier than everything above by orders of magnitude.

Playwright — drives a browser, better

Microsoft's more recent take on the same job. The significant practical difference is auto-waiting: Playwright waits for an element to actually be ready before interacting with it, which eliminates most of the flaky sleep(3) code that Selenium scripts accumulate. It also handles multiple browser contexts cleanly, which matters when you're running many scrapes at once.

For a new scraping project in 2026, Playwright is the better default. Selenium remains the right call when you're joining a codebase that already uses it, need a language Playwright doesn't serve as well, or need Selenium Grid's particular distributed setup.

Scrapy — a whole framework

Not a fetcher or a parser but an entire crawling framework: scheduling, concurrency, retries, deduplication, pipelines, throttling and export, with its own way of structuring a project. It's excellent at what it's for — crawling a lot of pages, politely and reliably — and it is far too much machinery for scraping one page.

Note that Scrapy doesn't run JavaScript on its own. For that it needs a plugin that hands the page to a browser, at which point you're running both stacks.

The decision, as a flow

Step one: is there an API? Check the documentation, and then check the page itself — see "the trick everyone misses" below. If there's an API, use it and stop reading.

Step two: does the data exist in the raw HTML? Run curl https://example.com or view-source and search for the value you want. If it's there, use requests + BeautifulSoup. Don't launch a browser.

Step three: is the data missing until JavaScript runs? Then you need a browser — Playwright by default. But do the trick below first, because half the time it lets you skip the browser anyway.

Step four: is it thousands of pages? Then you need crawling infrastructure — Scrapy, or your own queue and worker setup if you'd rather not adopt a framework.

The trick everyone misses

When a page builds itself with JavaScript, that JavaScript is fetching the data from somewhere — usually a JSON endpoint on the same site. You can call that endpoint directly.

Open your browser's developer tools, go to the Network tab, filter to Fetch/XHR, and reload the page. You'll often see exactly the data you want, as clean JSON, with no HTML to parse and no browser to run.

Here's a real comparison from the build sheet for this. Plain requests against a JavaScript-rendered page:

job elements found: 0
page text: JobsLoading…

The same page through Playwright, waiting for the content:

 - Backend Engineer | Remote
 - Data Analyst | Ottawa
JSON endpoints the page called: ['http://localhost:8899/jobs.json']

And that endpoint, hit directly with no browser at all:

 - Backend Engineer | Remote | $140k
 - Data Analyst | Ottawa | $95k

Note the salary field. The JSON contained data the rendered page never displayed — which is common, and a reminder to look before you build.

Side by side

The mistakes that cost people weeks

Launching a browser for a static page. The single most expensive habit in scraping. It turns a job that takes forty milliseconds into one that takes four seconds and needs a gigabyte of RAM.

Using r.text instead of r.content. When a server doesn't declare its character set, requests guesses — and guesses ISO-8859-1. Your prices come out as £1,299.00. Hand the raw bytes to your parser and let it read the page's own declared encoding. This bit us while testing the build sheet for this guide, which is why it's called out here.

Selecting on classes that are generated. .css-1x7ab3k is a build artefact and will change without warning. Prefer stable attributes, data- hooks, or structural selectors anchored to visible text.

No delay between requests. Hammering a site is how you get blocked, and it's rude. One or two seconds between requests costs you nothing on a small job.

Adopting Scrapy for six pages. You'll spend longer learning its project structure than writing the script it replaces.

Not saving the raw HTML. When your selector breaks in three weeks, having the original page saved makes the fix five minutes instead of an afternoon of guessing.

The legal and ethical part, briefly

Scraping publicly accessible data has generally been treated as lawful in the US — the long-running hiQ v. LinkedIn litigation is the case usually cited — but "generally lawful" is not "always fine", and several things genuinely do change the answer: data behind a login, personal data (GDPR applies regardless of whether the page was public), a site's terms of service, copyrighted content, and volumes heavy enough to affect the service. Rules differ by country, and obligations around AI training data have been tightening.

The practical standard that keeps you out of trouble: read robots.txt and honour it, identify yourself in a real User-Agent string with a contact, rate-limit deliberately, take only what you need, never scrape behind a login you agreed terms to get, and don't republish someone else's content as your own. If a project would be embarrassing to explain to the site's owner, that's the signal.

None of this is legal advice — if a project is commercially significant, ask someone qualified in your jurisdiction.

Now build one

Theory has a short shelf life here. The change watcher is requests plus BeautifulSoup, about sixty lines, and finishes in an evening. The Playwright build sheet covers the JavaScript case and walks through the Network-tab trick properly. Both include the code that actually ran, with output.