Extracting clean, readable article content from the web is deceptively hard. A page that looks simple in a browser is actually a dense forest of navigation bars, ads, sidebars, related links, and tracking scripts, with the actual article buried somewhere in the middle. Whether the goal is a read-it-later app, a translation pipeline, or a research tool, getting reliable article extraction right requires attention to a handful of key practices.
This guide covers the techniques and principles that separate fragile scrapers from robust extraction systems.
Start with the Right Tool for the Job
Not all scraping requires a full browser. For article extraction specifically, there are three tiers of complexity:
HTTP client + parser. For the majority of news sites, blogs, and publications, a simple HTTP request followed by HTML parsing is enough. Tools like Node.js’s fetch or Python’s requests library, combined with a parser like Cheerio or BeautifulSoup, handle these cases well. This approach is fast, lightweight, and easy to deploy.
Readability algorithms. Mozilla’s Readability library (and ports like @mozilla/readability for Node.js or readability-lxml for Python) are purpose-built for article extraction. They analyze the DOM structure, score content blocks by density of text versus markup, and return the article title, byline, and clean content. This is the natural default starting point. It handles 80-90% of articles correctly out of the box.
Headless browsers. Some modern sites render content entirely via JavaScript. Single-page applications, sites using heavy client-side rendering, or pages behind cookie consent walls may return empty or skeleton HTML to a plain HTTP request. For these, tools like Puppeteer or Playwright can render the page fully before extraction. This works best as a fallback, not a default. Headless browsers are slower, use more memory, and are harder to run at scale.
Respect the Site and Its Rules
Sustainable scraping means being a good citizen of the web.
Check robots.txt first. Before scraping any domain, the scraper should fetch and parse its robots.txt file, respecting Disallow directives and crawl-delay settings. This isn’t just about ethics. Ignoring robots.txt can get an IP blocked and, in some jurisdictions, create legal exposure.
Rate limit requests. Even if a site doesn’t specify a crawl delay, hammering a server with rapid requests is poor practice. We introduce delays between requests. One to two seconds per request is a reasonable baseline. For batch processing, a queue with configurable concurrency beats firing off all requests simultaneously.
Set a meaningful User-Agent. A descriptive User-Agent string that includes contact information or a URL identifies the scraper. This lets site operators reach us if there’s a problem, and it distinguishes our traffic from malicious bots. Spoofing browser User-Agent strings is best avoided unless there’s a specific technical reason (some sites serve different content to non-browser agents).
Handle errors gracefully. We expect and handle HTTP 429 (Too Many Requests), 403 (Forbidden), and 5xx errors, implementing exponential backoff for retries. If a site consistently blocks the scraper, that signal deserves respect.
Deal with HTML Structure Variability
The web is not standardized. Every site structures its HTML differently, and even a single site may use different templates for different article types.
Don’t rely on specific CSS selectors. A scraper built around .article-body > p will break the moment the site redesigns. Readability-style algorithms are more resilient because they work from general heuristics (text density, element scoring) rather than specific selectors.
Handle encoding correctly. Not every page is UTF-8. The Content-Type header and the HTML <meta charset> tag both need checking. Libraries like jsdom handle this automatically, but when parsing raw bytes, incorrect encoding will produce garbled text, especially for non-Latin scripts.
Strip boilerplate aggressively. Navigation, footers, related article links, social sharing buttons, and comment sections are noise. Readability handles most of this, but additional post-processing is sometimes needed. A good test: if the extracted text makes sense read aloud with no context about the site layout, the extraction is clean.
Preserve meaningful structure. Stripping noise is good, but flattening everything to plain text is not. Headings, lists, blockquotes, and emphasis carry meaning. We extract both a clean text version and an HTML version that preserves semantic markup. This gives downstream consumers flexibility.
Extract Metadata, Not Just Content
A well-extracted article is more than its body text. We capture:
- Title, from
<title>, Open Graph tags, or the Readability result - Author/byline, from byline elements,
<meta name="author">, or structured data - Publication date, from
<time>elements,article:published_timemeta tags, or JSON-LD - Excerpt/description, from meta description or Open Graph description
- Site name, from
og:site_nameor the domain itself - Canonical URL, from
<link rel="canonical">to avoid duplicate content from URL variations
Structured data (JSON-LD, Microdata) is increasingly common and is often the most reliable source for metadata. We check for <script type="application/ld+json"> blocks before falling back to meta tags.
Handle Edge Cases
Real-world article extraction means dealing with the messy edges:
Paywalled content. Many sites serve truncated content to non-subscribers. The extractor should detect this. If the extracted content is suspiciously short or ends with a “subscribe to read more” pattern, it is better to flag it than to silently return a partial article.
Multi-page articles. Some publications split articles across multiple pages. The pipeline can look for “next page” links or pagination patterns, then either follow them automatically or report the additional URLs for separate processing.
Non-article pages. Not every URL points to an article. The pipeline should handle landing pages, category pages, and error pages gracefully, detecting and reporting them rather than forcing extraction on content that isn’t an article.
Character encoding edge cases. Smart quotes, em dashes, and other typographic characters may be encoded differently across sites. Normalizing Unicode where appropriate resolves this.
Build for Observability
Article extraction at any scale needs monitoring:
- Log extraction quality signals. Track the ratio of extracted content length to raw HTML length. A very low ratio might indicate extraction failure; a very high ratio might mean boilerplate wasn’t stripped.
- Sample and review. Periodically spot-check extracted articles against their source pages. Automated extraction will drift as sites change their templates.
- Track failure rates by domain. If a particular site’s articles consistently fail extraction, a site-specific adapter or a different extraction strategy is likely needed for that domain.
Conclusion
Reliable article extraction is a practice, not a one-time implementation. We start with proven tools like Mozilla’s Readability, respect the sites we scrape, handle the inevitable edge cases, and build enough observability to catch problems before users do. The web changes constantly, and an extraction pipeline should be built to adapt with it.