Turning saved web pages into clean Markdown

ScrapePy saves websites for offline reading, and by default it writes a Markdown copy of every page beside the HTML. Saving websites for offline reading covers the HTML copy, and the frontmatter and dates at the top of each Markdown file. This post is about the body of those files.

Running a page through markdownify takes one line. On most sites, though, most of the result isn’t the page’s content. It’s the navigation, a cookie banner, a “Most read” list, share buttons, a newsletter box and a comment form, with the article somewhere in the middle. The converter lives in scrapepy/convert/to_markdown.py, and most of its code is about deciding what to leave out.

Not every file is a page

Before it converts anything, convert_file() skips files that aren’t worth a Markdown copy: RSS and Atom files the crawl saved, XML manifests, WordPress feed, author, category and tag pages, and error pages.

An error page is one whose title or main heading contains a marker such as “404”, “403” or “not found”, or whose body is short and says something like “page not found” or “access denied”. The check is blunt. The markers include “500” and “error”, so an article called “The 500 best albums” would be skipped as well.

Finding the content

extract_content() tries several extractors in turn, and takes the first result longer than 100 characters.

First come documentation platforms, which lay pages out in their own ways: API reference sites, Swagger UI, and help systems built with RoboHelp or MadCap Flare. They’re recognised by class names, ids and meta tags, and they have three extractors of their own.

Next is the standard extractor, for blogs and most other sites. It removes the junk described below, then tries a list of content selectors from the most specific to the least: .entry-content for WordPress, .post-content, .article-body, article, [role='main'], main, #content and a few more. The first one that matches more than 50 words wins:

for sel in selectors:
    content = soup.select_one(sel)
    if content and len(content.get_text(strip=True).split()) > 50:
        break
else:
    return None

The word count matters. Without it, a small element that happens to match, such as an empty main or a teaser card marked up as an article, would win over the real content further down the list.

If that finds nothing, trafilatura, a library built for pulling the main text out of a page, gets a go, with images, links and tables kept. The last resort strips the obvious furniture and converts whatever is in main, article or body.

The extractors change the parsed page as they go, removing elements they don’t want. So if any of the documentation extractors ran and gave up, the standard one parses the HTML afresh rather than working on what they left behind.

Removing junk before converting

Some junk is easiest to remove while it’s still HTML. _remove_junk_elements() deletes scripts, styles, nav, header and footer elements, anything with role="navigation", and aside or section elements whose class or id mentions a sidebar, widget or menu. Then it looks at every element’s class and id for any of 39 keywords, such as advert, newsletter, most-read, trending, social-share and cookie-consent:

for tag in list(soup.find_all(True)):
    if not tag.parent:
        continue  # Already decomposed with a parent element
    classes = " ".join(tag.get("class", []))
    tag_id = tag.get("id", "")
    combined = (classes + " " + tag_id).lower()
    if any(kw in combined for kw in _JUNK_CLASS_KEYWORDS):
        tag.decompose()

The loop runs over a list taken before anything is removed, so it still reaches the children of an element it has already deleted. Those have no parent any more, and the check at the top skips them. A test puts an ad-wrapper inside an ad-container to make sure that case doesn’t fail. The keywords match anywhere in the class or id, which is blunt too: any element whose class contains “subscribe” goes, whatever it holds.

Cleaning the Markdown

Other junk only shows up as text: “Advertisement” on a line of its own, “3 hours ago” under the headline, “Leave this field blank” from a newsletter form, share counts, “Skip to main content”. clean_markdown() removes these with a long run of regular expressions, grouped into inline patterns, whole sections, larger blocks and structural fixes.

Most of the inline patterns match a whole line, from ^ to $, so the same words inside a sentence survive. One test checks that a timestamp on its own line goes, and that “He said this happened 2 hours ago during the meeting.” keeps its “2 hours ago”.

The section patterns remove a heading such as “Most Read”, “Related Articles”, “Trending”, “You May Also Like” or “Comments”, with everything under it up to the next heading of level 1 to 3, or the end of the file. But plenty of sites mark those labels up as a bold paragraph rather than a heading, and a bold line doesn’t match a heading pattern. So convert_file() turns bold lines into headings first:

# Promote bold lines to headings first so junk section patterns
# (which match ## headings) can catch promoted junk headings
md = self.promote_bold_headings(md)

md = self.clean_markdown(md)

A line is promoted when it’s nothing but bold text, under 80 characters, with no full stop before its end. That keeps a bold sentence in the middle of an article as a sentence.

Many of the patterns target the markup of one particular site. The tests are grouped by site, each group with samples of that site’s junk, and checks that the real text around it survives.

Structural fixes

The last group tidies what’s left. A site name is stripped from the end of the main heading, so “# Title | Example.com” becomes “# Title”. One pattern catches names that look like a domain. Another catches a short brand name after a separator, but only when the title before it is at least 20 characters long, so a heading like “Results | Round 5” survives. There’s a test for that one.

A repeated copy of the main heading is removed, as are an image that appears twice in a row, empty headings, empty links and copyright lines. Link text that runs across several lines is joined into one, and runs of blank lines are collapsed. If 50 characters or fewer are left, the page is dropped. A page whose cleaned text runs to fewer than 50 words is dropped too, as a list or section page with no content of its own.

Links to the Markdown copy

By this point the saved HTML’s links already point at other saved files, such as news/story.html, because the HTML was rewritten for offline reading. convert_links() moves them over to the Markdown copy. A link ending in .html or .htm gets .md instead, a link ending in a slash becomes the folder’s name plus .md, and a link with no extension gets .md added. Full URLs, anchors, mailto: and tel: links are left alone. A link to the page itself, or to an index page in the same folder or any folder above it, becomes plain text.

Using it

pip install .
scrapling install
scrapepy https://docs.example.com/guide/ --stay-in-path

The HTML goes into ScrapedSites/ and the Markdown into MarkdownSites/. Add --no-markdown to keep only the HTML.

If you use it, follow each site’s terms, respect its robots.txt and rate limits, and only keep what you have the right to keep. The code is on GitHub.