Web Scraping Pipelines

2. Data Scraping & Web Pipelines

This section covers how to download public web pages, read their structural code (the DOM tree), and pull out clean text or links.

Tools & Libraries

• requests: A standard, user-friendly tool used to send network requests to web pages to download their raw HTML text.
• BeautifulSoup4 (bs4): A parsing engine that reads messy HTML web code and organizes it into a searchable, navigable tree.
• html.parser: The built-in, batteries-included engine Python uses out of the box to read and parse HTML files.

---

Industry Upgrades (What Teams Use at Scale)

While requests and BeautifulSoup are great for smaller tasks, production environments dealing with millions of pages upgrade to asynchronous or framework-based tools.

1. Async Fetching: httpx & aiohttp

Instead of waiting for one page to finish downloading before starting the next (synchronous), these libraries allow your script to send hundreds of requests at the same time (asynchronous), drastically speeding up your pipeline.

• How it looks in code (httpx example):

import httpx
import asyncio

async def fetch_page(url):
    # 'async with' lets Python pause this specific task while waiting for the network,
    # allowing other URLs to download in the background.
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        return response.status_code

# Running it asynchronously
# status = asyncio.run(fetch_page('[https://httpbin.org/get](https://httpbin.org/get)'))

2. The Full Pipeline Framework: Scrapy

Scrapy isn't just a library; it's a powerful, enterprise-grade web scraping framework built to crawl massive websites with built-in data cleaning.

• How it looks in code:

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['[https://news.ycombinator.com/](https://news.ycombinator.com/)']

    def parse(self, response):
        # Scrapy combines fetching and CSS extraction directly into one clean pipeline
        for title in response.css('.titleline > a'):
            yield {
                'title': title.css('::text').get(),
                'link': title.css('::attr(href)').get()
            }
---

Core Methods & Code Example

• requests.get(url, headers=headers): Downloads the raw HTML code from a website. We add custom User-Agent headers to make our script look like a real web browser so websites don't block us.
• BeautifulSoup(response.text, 'html.parser'): Converts raw web code into an interactive layout map (Document Object Model) that our script can easily read.
• soup.select('.class_name') / soup.find_all('tag'): Uses CSS design selectors or direct HTML tags to pinpoint and grab exactly what you want (like targeting specific article headlines).
• .get_text(strip=True) & .get('href'): Strips away the ugly web code tags, leaving behind clean string text or extracting web link URLs.

import requests
from bs4 import BeautifulSoup

url = "[https://news.ycombinator.com/](https://news.ycombinator.com/)"
fake_browser_header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}

# Fetch the raw website content safely
response = requests.get(url, headers=fake_browser_header)

# Convert raw text into a searchable DOM tree layout map
soup = BeautifulSoup(response.text, 'html.parser')

# Use CSS Selectors to target precise structural tags (.titleline)
articles = soup.select('.titleline > a')
for link in articles:
    text = link.get_text(strip=True) # Clear away ugly web tag code
    href = link.get('href')          # Isolate the explicit hyperlink attribute
    print(f"Headline: {text} | Link: {href}")