Web scraping divides developers: some see it as a power tool for gathering data at scale, others as a gray-area practice with legal and ethical landmines. In 2026, both perspectives are correct. The right approach depends on what you are scraping, whose data it is, and how you are doing it. Here is the complete practical guide.
What changed in 2026
- Anti-bot systems are mainstream. Cloudflare Turnstile, DataDome, and PerimeterX are on most major sites. Getting blocked is the default outcome for naive scrapers, not an edge case.
- Playwright replaced Selenium as the standard browser automation library — faster, async-native, and maintained by Microsoft with excellent Python bindings.
- AI-assisted scraping is real. Tools like Firecrawl and Crawl4AI use LLMs to understand page structure and extract data without writing CSS selectors.
httpx largely replaced requests for HTTP work — async support, HTTP/2, and cleaner API.robots.txt enforcement became more legally significant — courts in multiple jurisdictions upheld ToS-based claims against scrapers in 2024–2025.
When to scrape vs use an API
| Situation |
Do this |
| Site has a public REST/GraphQL API |
Use the API — more stable, legal, faster |
| Data is licensed (financials, real estate) |
Buy the data feed |
| Public data, no API, static HTML |
Scrape with httpx + BS4 |
| Public data, JS-rendered SPA |
Scrape with Playwright |
| Logged-in data you own |
Playwright with your credentials |
| Competitor prices / proprietary data |
Consult a lawyer first |
Static scraping: httpx + BeautifulSoup
import httpx
from bs4 import BeautifulSoup
import time
HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; MyResearchBot/1.0; +https://example.com/bot)"
}
def scrape_page(url: str) -> list[dict]:
response = httpx.get(url, headers=HEADERS, follow_redirects=True, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, "lxml")
items = []
for article in soup.select("article.post"):
title = article.select_one("h2.title")
link = article.select_one("a[href]")
date = article.select_one("time[datetime]")
items.append({
"title": title.get_text(strip=True) if title else None,
"url": link["href"] if link else None,
"date": date["datetime"] if date else None,
})
return items
# Be polite — respect the server
for page in range(1, 11):
data = scrape_page(f"https://example.com/posts?page={page}")
time.sleep(1.5) # 1.5s between requests
JavaScript-rendered sites: Playwright
import asyncio
from playwright.async_api import async_playwright
async def scrape_spa(url: str) -> list[str]:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
# Set a real user agent
await page.set_extra_http_headers({
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
})
await page.goto(url, wait_until="networkidle")
# Wait for dynamic content
await page.wait_for_selector(".product-card", timeout=10_000)
# Extract text from all matching elements
titles = await page.eval_on_selector_all(
".product-card h3",
"els => els.map(el => el.textContent.trim())"
)
await browser.close()
return titles
titles = asyncio.run(scrape_spa("https://example.com/products"))
Tool comparison for 2026
| Tool |
Best for |
Handles JS |
Speed |
Difficulty |
| httpx + BS4 |
Static HTML |
No |
Fast |
Low |
| Playwright |
SPAs, login, clicks |
Yes |
Slow |
Medium |
| Scrapy |
Large-scale crawls |
No (without plugin) |
Fast |
High |
| Firecrawl |
LLM-powered extraction |
Yes |
Medium |
Low |
| Selenium |
Legacy browser automation |
Yes |
Very slow |
Medium |
Respecting rate limits and being polite
import asyncio
import random
async def polite_fetch(client: httpx.AsyncClient, url: str) -> str:
# Random delay between 1–3 seconds — less fingerprint-able than fixed
await asyncio.sleep(random.uniform(1.0, 3.0))
response = await client.get(url)
response.raise_for_status()
return response.text
Good scraper hygiene:
- Identify yourself in the User-Agent string.
- Respect
Crawl-delay in robots.txt.
- Cache responses — do not re-fetch pages you already have.
- Scrape during off-peak hours for the target site.
- Limit concurrent requests to 2–5 max.
How to handle common blocking patterns
| Block type |
Signal |
Fix |
| Rate limit |
429 Too Many Requests |
Back off exponentially, reduce concurrency |
| IP block |
403 / connection refused |
Rotate IPs (ethically); check if you are violating ToS |
| CAPTCHA |
Challenge page in response |
Use Playwright with a CAPTCHA-solving service or stop |
| JS challenge |
Cloudflare interstitial |
Use Playwright with stealth mode plugin |
| Session expiry |
Redirect to login |
Handle cookies / re-authenticate |
How to pick the right scraping approach
- Is the data available via an API? Use it. Scrapers break when sites redesign; APIs have versioning.
- Is the content static HTML? httpx + BS4 is 10x faster than a browser and sufficient.
- Does the page require JavaScript execution? Use Playwright.
- Are you scraping 100k+ pages? Use Scrapy with its middleware, retry, and pipeline system.
- Do you need to extract unstructured text? Consider Firecrawl or Crawl4AI for LLM-assisted extraction.
Common mistakes
Not handling pagination. Many scrapers grab page 1 and stop. Detect next-page links or build page-number ranges explicitly.
Storing raw HTML instead of parsed data. Parse before storing — raw HTML has noise, changes format, and is expensive to re-parse later.
No retry on transient errors. Network timeouts and 5xx errors are temporary. Use tenacity or httpx's retry transport.
Scraping at full concurrency. 50 concurrent requests to a small site causes real server harm and guaranteed blocks. Limit to 2–5 concurrent.
Ignoring encoding. Non-UTF-8 pages will give garbled data. Always use response.encoding or detect with chardet.
What to skip
- Selenium in 2026 — Playwright is faster, has better async support, and has better selectors. There is no reason to start new projects with Selenium.
- Building your own proxy rotation system — use a managed proxy provider if you legitimately need IP rotation; DIY systems are fragile.
- Scraping rate-limited APIs through their web UI — violates ToS and is outpaced by the API rate limit anyway.
FAQ
Is web scraping legal in 2026?
It depends on jurisdiction, what data you are scraping, and how you use it. Publicly available data scraped non-intrusively is generally legal in most jurisdictions. Violating explicit ToS, scraping personal data under GDPR, or scraping behind authentication creates real legal risk. When uncertain, consult a lawyer.
How do I avoid getting blocked?
Identify yourself honestly, respect robots.txt, rate-limit your requests, use realistic headers, and handle 429 responses with backoff. Trying to disguise scraping as human traffic is both an arms race and a ToS violation.
What is the best library for large-scale crawling in 2026?
Scrapy remains the best for crawling hundreds of thousands of pages — it handles deduplication, retries, pipelines, and distributed crawls. For smaller projects, httpx with asyncio is simpler.
How do I extract data from PDFs or images on a scraped page?
Download the file with httpx, then use pdfplumber or pypdf for PDFs, or an OCR library like pytesseract / a vision LLM API for images.
Where to go next