Saving websites for offline reading with ScrapePy

Saving a single web page is easy. Saving a whole site so that it still works with no connection is harder. Every page has to be on disk, every image with it, and every link has to point at the copy instead of the live site. ScrapePy is a Python tool I wrote for this, published on 18 October 2026. Here are the parts that make an offline copy work.

Look before crawling

Before it fetches any content, ScrapePy runs a short reconnaissance step. It reads robots.txt to find the sitemaps, reads the sitemaps and any RSS or Atom feeds to build a map of URLs and their dates, and parses the home page’s header, navigation and footer, including dropdown menus. It saves the site’s section structure to site_outline.json and starts the crawl from the navigation rather than just the home page. On a large site that means the sections are found early, and with date filtering on, old articles can be skipped without being fetched.

One URL, one file

An offline copy needs a fixed rule for where each URL lives on disk. /about becomes about.html, / becomes index.html, and anything with an extension keeps it. Two details matter.

News sites often have numeric IDs in their paths. Mapped literally, /news/12345/1/slug would become three nested folders holding one file. ScrapePy folds numeric segments next to the file name into it instead, so that URL becomes news/slug-12345-1.html.

URLs come from the site being copied, so they can’t be trusted. After mapping, the path is normalised and checked: if it would land outside the output folder, it falls back to a flat file name.

result = os.path.normpath(os.path.join(config.output_dir, clean))

output_dir_resolved = os.path.normpath(config.output_dir)
if not result.startswith(output_dir_resolved + os.sep) and result != output_dir_resolved:
    safe_name = clean.replace("/", "_").replace("\\", "_")
    result = os.path.join(config.output_dir, safe_name)

Rewriting the links

Once a page is saved, its links still point at the live site. The rewriter goes through every href and src, decides whether it’s internal, and points internal ones at the copy with a path relative to the current file.

Deciding what’s internal is the fiddly part. Fragments, mailto:, tel:, javascript: and data: links are left alone. Root-relative paths and URLs starting with the site’s base URL are internal. Any other absolute URL is external. Anything else is a relative path, and so internal.

Internal links are then sorted by type. Stylesheets point into css/, scripts into js/, images into images/, and documents such as PDFs into media/. Pages point at their mapped HTML file. With --stay-in-path, a page outside the starting path points back to the live site instead of to a file that was never downloaded.

Only fetch what changed

Copies go out of date, and re-downloading a whole site to refresh it is slow and puts needless load on the server. So ScrapePy keeps a small cache file in each copy, with the ETag, Last-Modified value and a SHA-256 hash of every page.

On the next run it sends conditional request headers:

def get_page_headers(self, url_path: str) -> dict:
    entry = self._data["pages"].get(url_path)
    if not entry:
        return {}
    headers = {}
    if entry.get("etag"):
        headers["If-None-Match"] = entry["etag"]
    if entry.get("last_modified"):
        headers["If-Modified-Since"] = entry["last_modified"]
    return headers

If the server answers 304 Not Modified, the page is skipped. The crawl still needs the links on that page, though, or it would never reach the pages behind it, so the cache stores each page’s links and replays them. Plenty of servers don’t send these headers at all. For those, ScrapePy fetches the page and compares its hash with the stored one, and skips the write if nothing changed.

A Markdown copy too

Alongside the HTML, ScrapePy writes a Markdown copy of the site, which is easier to search, diff or feed into other tools. Each file starts with YAML frontmatter: title, URL, date, page type, word count and description. The image folders are copied across so the Markdown’s image links still work.

The date takes some care. ScrapePy looks for it in several places and ranks them by how much it trusts each one: a meta tag first, then JSON-LD, then a date in the URL, then a <time> element, and the HTTP headers last. The frontmatter records which source it used.

Using it

pip install .
scrapling install
scrapepy https://docs.example.com/guide/ --stay-in-path

The HTML copy goes into ScrapedSites/ and the Markdown into MarkdownSites/. Run the same command again later and unchanged pages are skipped.

If you use it, follow each site’s terms, respect its robots.txt and rate limits, and only keep what you have the right to keep. The code is on GitHub.