Every bot library, grouped by the job it does
Libraries get compared in flat lists that mix tools doing completely different jobs. Here they're grouped by function, so you can see what's actually an alternative to what.
Nearly every bot is assembled from the same eight groups. Pick one tool from each group you need and you have a working system. The mistake is comparing across groups — a fetcher isn't an alternative to a parser.
1. Fetching — getting bytes from a server
| Tool | Language | Reach for it when |
|---|---|---|
| requests | Python | The default. Synchronous, readable, everywhere. |
| httpx | Python | You need async, HTTP/2, or to fetch hundreds of URLs concurrently. |
| aiohttp | Python | You're already in an asyncio codebase. |
| curl-cffi | Python | A site fingerprints TLS handshakes and blocks ordinary clients. |
| fetch / undici | JavaScript | You're in Node. fetch is built in now. |
Start with requests. Move to httpx only when concurrency is the actual bottleneck — which is later than you think.
2. Parsing — finding things inside HTML
| Tool | Language | Reach for it when |
|---|---|---|
| BeautifulSoup | Python | The default. Tolerant of malformed HTML, pleasant API. |
| lxml | Python | Speed matters, or you want real XPath. Also BeautifulSoup's best engine. |
| selectolax | Python | Parsing at serious volume; noticeably faster than both. |
| Cheerio | JavaScript | Node, and you want jQuery-style selectors. |
Install lxml alongside BeautifulSoup and pass "lxml" as the parser — same API, considerably faster than the built-in one.
3. Driving a browser — when the page needs to run
| Tool | Reach for it when |
|---|---|
| Playwright | The default for new work. Auto-waiting removes most flakiness; clean multi-context support; Python, JS, .NET, Java. |
| Selenium | Existing Selenium codebase, Selenium Grid, or a language where it's better supported. |
| Puppeteer | Node-only projects, Chrome-only. Playwright grew out of this lineage. |
| Patchright / undetected drivers | A site actively detects automation. Understand the terms you're agreeing to before you go here. |
Every tool in this group is roughly a hundred times more expensive per page than groups 1 and 2. Confirm you need it.
4. Crawling — many pages, politely
| Tool | Reach for it when |
|---|---|
| Scrapy | Thousands of pages. Gives you scheduling, retries, throttling, deduplication and pipelines for free. |
| Crawlee | Same idea in Node/Python, with browser and HTTP crawling under one API. |
| Your own queue | Under a few hundred pages. A list, a set of seen URLs and a sleep is genuinely enough. |
5. Scheduling — making it run without you
| Tool | Reach for it when |
|---|---|
| cron | A machine that's reliably on. Remember it doesn't inherit your shell's environment variables — the classic silent failure. |
| GitHub Actions (schedule) | Free, always on, versioned. Excellent for anything hourly or slower. |
| systemd timers | A Linux box or a Raspberry Pi you control, and you want restart-on-failure. |
| APScheduler / Celery beat | Scheduling belongs inside a long-running Python application. |
6. Talking to platforms
| Platform | Library | Note |
|---|---|---|
| Telegram | python-telegram-bot | Easiest platform to start on — a token from @BotFather and you're live. |
| Discord | discord.py | Mature. Mind the gateway intents, which trip everyone up once. |
| Slack | slack-bolt | Socket Mode saves you exposing a public URL. |
| Cloud API / Twilio | Requires business verification and template approval. Budget days, not hours. | |
| imaplib / Gmail API | IMAP for reading anything; the Gmail API when you need labels and threading. | |
| Anywhere | A plain webhook | Often all you need. Don't add an SDK for one HTTP POST. |
7. Storing what you collected
| Tool | Reach for it when |
|---|---|
| SQLite | Almost always, to start. One file, no server, handles millions of rows, ships with Python. |
| CSV / Parquet | The output is a dataset for someone else. Parquet once it's large. |
| Postgres | Several things write concurrently, or you've outgrown one file. |
| pandas | Cleaning and reshaping between scrape and storage. |
Store the raw response as well as the parsed fields. When a selector breaks, the original page is what lets you fix it quickly.
8. Staying unblocked
Before any tooling: a real User-Agent that identifies you and offers a contact, a deliberate delay between requests, honouring robots.txt, and caching so you never fetch the same page twice. That alone keeps most well-behaved scrapers welcome.
Beyond that sit rotating proxies, residential IP pools and CAPTCHA-solving services. They exist, they work, and they're also the point at which you're actively circumventing a site's stated wishes — which changes the ethical and sometimes legal position. Worth being deliberate rather than drifting there.
Two starter stacks
Watch a few pages for changes: requests + BeautifulSoup + lxml + SQLite + GitHub Actions. Free to run, finishes in an evening. That's exactly the change watcher.
Scrape a JavaScript-heavy site: check the Network tab for a JSON endpoint first; if there isn't one, Playwright + SQLite. That build sheet walks both routes.
One new build sheet a week
Guides like this one, plus a new build sheet every Thursday. Written by a person who built it.