{"post":{"slug":"how-to-scrape-a-website-2026","title":"How to scrape a website in 2026: tools, legality, and best practices","excerpt":"Web scraping in 2026 is harder than it was five years ago — anti-bot systems are sophisticated. But the right tools and ethical approach still get the job done reliably.","category":"code","date":"April 2, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">W</span>eb scraping divides developers: some see it as a power tool for gathering data at scale, others as a gray-area practice with legal and ethical landmines. In 2026, both perspectives are correct. The right approach depends on what you are scraping, whose data it is, and how you are doing it. Here is the complete practical guide.\n\n## What changed in 2026\n\n- **Anti-bot systems are mainstream.** Cloudflare Turnstile, DataDome, and PerimeterX are on most major sites. Getting blocked is the default outcome for naive scrapers, not an edge case.\n- **Playwright replaced Selenium as the standard browser automation library** — faster, async-native, and maintained by Microsoft with excellent Python bindings.\n- **AI-assisted scraping is real.** Tools like Firecrawl and Crawl4AI use LLMs to understand page structure and extract data without writing CSS selectors.\n- **`httpx` largely replaced `requests`** for HTTP work — async support, HTTP/2, and cleaner API.\n- **`robots.txt` enforcement became more legally significant** — courts in multiple jurisdictions upheld ToS-based claims against scrapers in 2024–2025.\n\n## When to scrape vs use an API\n\n| Situation | Do this |\n| --- | --- |\n| Site has a public REST/GraphQL API | Use the API — more stable, legal, faster |\n| Data is licensed (financials, real estate) | Buy the data feed |\n| Public data, no API, static HTML | Scrape with httpx + BS4 |\n| Public data, JS-rendered SPA | Scrape with Playwright |\n| Logged-in data you own | Playwright with your credentials |\n| Competitor prices / proprietary data | Consult a lawyer first |\n\n## Static scraping: httpx + BeautifulSoup\n\n```python\nimport httpx\nfrom bs4 import BeautifulSoup\nimport time\n\nHEADERS = {\n    \"User-Agent\": \"Mozilla/5.0 (compatible; MyResearchBot/1.0; +https://example.com/bot)\"\n}\n\ndef scrape_page(url: str) -> list[dict]:\n    response = httpx.get(url, headers=HEADERS, follow_redirects=True, timeout=10)\n    response.raise_for_status()\n\n    soup = BeautifulSoup(response.text, \"lxml\")\n    items = []\n\n    for article in soup.select(\"article.post\"):\n        title = article.select_one(\"h2.title\")\n        link  = article.select_one(\"a[href]\")\n        date  = article.select_one(\"time[datetime]\")\n\n        items.append({\n            \"title\": title.get_text(strip=True) if title else None,\n            \"url\":   link[\"href\"] if link else None,\n            \"date\":  date[\"datetime\"] if date else None,\n        })\n\n    return items\n\n# Be polite — respect the server\nfor page in range(1, 11):\n    data = scrape_page(f\"https://example.com/posts?page={page}\")\n    time.sleep(1.5)   # 1.5s between requests\n```\n\n## JavaScript-rendered sites: Playwright\n\n```python\nimport asyncio\nfrom playwright.async_api import async_playwright\n\nasync def scrape_spa(url: str) -> list[str]:\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(headless=True)\n        page = await browser.new_page()\n\n        # Set a real user agent\n        await page.set_extra_http_headers({\n            \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\"\n        })\n\n        await page.goto(url, wait_until=\"networkidle\")\n\n        # Wait for dynamic content\n        await page.wait_for_selector(\".product-card\", timeout=10_000)\n\n        # Extract text from all matching elements\n        titles = await page.eval_on_selector_all(\n            \".product-card h3\",\n            \"els => els.map(el => el.textContent.trim())\"\n        )\n\n        await browser.close()\n        return titles\n\ntitles = asyncio.run(scrape_spa(\"https://example.com/products\"))\n```\n\n## Tool comparison for 2026\n\n| Tool | Best for | Handles JS | Speed | Difficulty |\n| --- | --- | --- | --- | --- |\n| httpx + BS4 | Static HTML | No | Fast | Low |\n| Playwright | SPAs, login, clicks | Yes | Slow | Medium |\n| Scrapy | Large-scale crawls | No (without plugin) | Fast | High |\n| Firecrawl | LLM-powered extraction | Yes | Medium | Low |\n| Selenium | Legacy browser automation | Yes | Very slow | Medium |\n\n## Respecting rate limits and being polite\n\n```python\nimport asyncio\nimport random\n\nasync def polite_fetch(client: httpx.AsyncClient, url: str) -> str:\n    # Random delay between 1–3 seconds — less fingerprint-able than fixed\n    await asyncio.sleep(random.uniform(1.0, 3.0))\n    response = await client.get(url)\n    response.raise_for_status()\n    return response.text\n```\n\nGood scraper hygiene:\n- Identify yourself in the User-Agent string.\n- Respect `Crawl-delay` in `robots.txt`.\n- Cache responses — do not re-fetch pages you already have.\n- Scrape during off-peak hours for the target site.\n- Limit concurrent requests to 2–5 max.\n\n## How to handle common blocking patterns\n\n| Block type | Signal | Fix |\n| --- | --- | --- |\n| Rate limit | 429 Too Many Requests | Back off exponentially, reduce concurrency |\n| IP block | 403 / connection refused | Rotate IPs (ethically); check if you are violating ToS |\n| CAPTCHA | Challenge page in response | Use Playwright with a CAPTCHA-solving service or stop |\n| JS challenge | Cloudflare interstitial | Use Playwright with stealth mode plugin |\n| Session expiry | Redirect to login | Handle cookies / re-authenticate |\n\n## How to pick the right scraping approach\n\n1. **Is the data available via an API?** Use it. Scrapers break when sites redesign; APIs have versioning.\n2. **Is the content static HTML?** httpx + BS4 is 10x faster than a browser and sufficient.\n3. **Does the page require JavaScript execution?** Use Playwright.\n4. **Are you scraping 100k+ pages?** Use Scrapy with its middleware, retry, and pipeline system.\n5. **Do you need to extract unstructured text?** Consider Firecrawl or Crawl4AI for LLM-assisted extraction.\n\n## Common mistakes\n\n**Not handling pagination.** Many scrapers grab page 1 and stop. Detect next-page links or build page-number ranges explicitly.\n\n**Storing raw HTML instead of parsed data.** Parse before storing — raw HTML has noise, changes format, and is expensive to re-parse later.\n\n**No retry on transient errors.** Network timeouts and 5xx errors are temporary. Use `tenacity` or `httpx`'s retry transport.\n\n**Scraping at full concurrency.** 50 concurrent requests to a small site causes real server harm and guaranteed blocks. Limit to 2–5 concurrent.\n\n**Ignoring encoding.** Non-UTF-8 pages will give garbled data. Always use `response.encoding` or detect with `chardet`.\n\n## What to skip\n\n- **Selenium in 2026** — Playwright is faster, has better async support, and has better selectors. There is no reason to start new projects with Selenium.\n- **Building your own proxy rotation system** — use a managed proxy provider if you legitimately need IP rotation; DIY systems are fragile.\n- **Scraping rate-limited APIs through their web UI** — violates ToS and is outpaced by the API rate limit anyway.\n\n## FAQ\n\n**Is web scraping legal in 2026?**\nIt depends on jurisdiction, what data you are scraping, and how you use it. Publicly available data scraped non-intrusively is generally legal in most jurisdictions. Violating explicit ToS, scraping personal data under GDPR, or scraping behind authentication creates real legal risk. When uncertain, consult a lawyer.\n\n**How do I avoid getting blocked?**\nIdentify yourself honestly, respect `robots.txt`, rate-limit your requests, use realistic headers, and handle 429 responses with backoff. Trying to disguise scraping as human traffic is both an arms race and a ToS violation.\n\n**What is the best library for large-scale crawling in 2026?**\nScrapy remains the best for crawling hundreds of thousands of pages — it handles deduplication, retries, pipelines, and distributed crawls. For smaller projects, `httpx` with `asyncio` is simpler.\n\n**How do I extract data from PDFs or images on a scraped page?**\nDownload the file with httpx, then use `pdfplumber` or `pypdf` for PDFs, or an OCR library like `pytesseract` / a vision LLM API for images.\n\n## Where to go next\n\n- [How to work with APIs in 2026](/blog/how-to-work-with-apis-2026)\n- [How to parse JSON in 2026](/blog/how-to-parse-json-2026)\n- [How to write a Python script in 2026](/blog/how-to-write-a-python-script-2026)","hue":310,"metaTitle":"How to Scrape a Website in 2026: Tools and Best Practices","metaDescription":"Complete 2026 guide to web scraping. Covers httpx, BeautifulSoup, Playwright, handling anti-bot systems, legal considerations, and building reliable","keywords":["how to scrape a website 2026","web scraping python 2026","playwright scraping tutorial","beautifulsoup tutorial 2026","httpx web scraping","web scraping best practices","web scraping legality 2026","anti-bot bypass 2026","scrapy vs playwright 2026","python web scraper guide"],"tldr":["**Check the robots.txt and terms of service first** — scraping data that violates ToS creates legal and ethical risk; use official APIs when available.","**Use `httpx` + `BeautifulSoup` for static sites** — lightweight, fast, and sufficient for any site that renders HTML server-side.","**Use Playwright for JavaScript-rendered sites** — it controls a real browser and handles SPAs, infinite scroll, and login flows.","**Respect rate limits and add delays** — hammering a server with concurrent requests gets you blocked and may cause harm; be a polite scraper.","**Skip:** building a custom scraper for data that has a public API or a licensed data feed — maintain cost vs build cost almost always favors the API."],"readTime":"5 min read","wordCount":1181},"related":[{"slug":"how-to-write-a-python-script-2026","title":"How to write a Python script in 2026: structure, tooling, and best practices","excerpt":"Writing a Python script that works once is easy. Writing one that other people can run, debug, and maintain in 2026 requires knowing the right structure, tooling, and modern idioms.","category":"code","date":"March 18, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":309},{"slug":"how-to-write-a-bash-script-2026","title":"How to write a Bash script in 2026: the safe and practical guide","excerpt":"Bash scripting is still the fastest way to automate repetitive shell work, but most scripts are written unsafely. Here is the 2026-correct approach.","category":"code","date":"April 6, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":306},{"slug":"how-to-write-e2e-tests-2026","title":"How to write e2e tests in 2026","excerpt":"End-to-end tests catch what unit tests miss — broken flows, real browser quirks, and integration failures. This guide covers Playwright setup, writing reliable tests, and running them in CI.","category":"code","date":"March 18, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":316}],"sameCategory":[{"slug":"bazel-build-explained-2026","title":"Bazel Explained: What Hermetic Builds Actually Cost","excerpt":"Bazel delivers reproducible, precisely cached builds and demands that you describe everything. Here is what you get, what it costs, and who should say no.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":145},{"slug":"graphql-federation-2026","title":"GraphQL Federation: One Graph, Many Teams","excerpt":"Federation composes independently-owned subgraphs into a single API. Here is what it solves, the operational cost it adds, and when a simpler gateway is enough.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":300},{"slug":"hermetic-builds-explained-2026","title":"Hermetic Builds Explained: Same Inputs, Same Output","excerpt":"A hermetic build depends only on declared inputs. Here is what that property gives you, the common leaks that break it, and how to move toward it incrementally.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":130},{"slug":"monorepo-tooling-compared-2026","title":"Monorepo Tooling Compared: What You Actually Need","excerpt":"Monorepo tools solve task orchestration and caching, not repository layout. Here is what each tier provides and how to tell when you have outgrown the simple option.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150}],"newer":{"slug":"how-to-say-no-politely-2026","title":"How to say no politely in 2026: scripts that work without burning bridges","excerpt":"Saying no is a professional skill, not a character flaw. This 2026 guide gives you concrete scripts and frameworks for declining without damaging relationships.","category":"guides","date":"April 2, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":150},"older":{"slug":"how-to-secure-your-router-2026","title":"How to secure your router in 2026: the checklist every homeowner needs","excerpt":"Your router is the front door of your home network, and most people leave it factory-configured. Here is the 2026 checklist to lock it down properly.","category":"tech","date":"April 2, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":248}}