---
title: ScrapePy
canonical: 'https://www.dealmobile.co.uk/projects/scrapepy'
url: 'https://www.dealmobile.co.uk/projects/scrapepy'
type: Project
summary: 'A command-line crawler that downloads a site with its images, rewrites the links so it works offline, and writes a Markdown copy with frontmatter. Re-runs only fetch what has changed.'
---

# ScrapePy

A Python web scraper that saves whole sites for offline reading and can turn them into Markdown.

![A Markdown file written by ScrapePy: its frontmatter with title, URL, date, page type, word count and domain highlighted, followed by the page’s heading and first link.](https://www.dealmobile.co.uk/sites/default/files/styles/almanac_shot_md/public/content/images/products/scrapepy-light.png.webp?itok=hr_0B8zs)

- **Kind:** Tool
- **Status:** Maintained
- **Started:** 2026
- **Licence:** MIT
- **Stack:** Python, Scrapling, BeautifulSoup, pytest
- **Links:** [Source on GitHub](https://github.com/tomgould/ScrapePy)

ScrapePy is a command-line tool for keeping a copy of a website: documentation you want to read on a train, a reference site you want to search locally, or articles you want to feed into your own tools as Markdown. You give it a URL. It crawls the site, saves every page with its images, and rewrites the links so the copy works in a browser with no connection.

## What it does

- Crawls every page on the same domain, or only pages under a starting path.
- Saves the HTML and the images, and by default the CSS, JavaScript and fonts too. It rewrites internal links to relative paths, so pages link to each other on disk.
- Writes a Markdown copy of the site alongside, with YAML frontmatter on every file: title, URL, date, page type, word count and description. Images are copied across so the Markdown links work.
- Re-runs incrementally. It sends the `ETag` and `Last-Modified` values from the last crawl, skips pages the server says haven’t changed, and compares a SHA-256 hash of the content for servers that don’t send those headers.
- Records broken links in a text file.
- Reads `robots.txt`, sitemaps and RSS or Atom feeds before crawling, and parses the home page’s navigation, including dropdown menus, to map the site’s sections first.
- Can keep only recent articles on news sites, for example the last two weeks, while keeping every section page.

It also has modes for single-page apps, documentation platforms and Swagger or OpenAPI references. An audit mode looks at a site’s structure without crawling it and saves test fixtures.

## How it’s built

ScrapePy needs Python 3.11 or later. Fetching uses [Scrapling](https://github.com/D4Vinci/Scrapling): plain HTTP requests for ordinary pages, and a real Chromium browser for pages that need JavaScript to render. Parsing uses BeautifulSoup and lxml, asset downloads run concurrently with httpx, and the Markdown comes from markdownify with trafilatura as a fallback for pulling out the main content.

A few details matter for offline copies. URLs are mapped to files carefully: `/about` becomes `about.html`, and a path that tries to climb out of the output folder falls back to a flat file name instead. Numeric path segments are folded into the file name, so `/news/12345/1/slug` becomes `news/slug-12345-1.html` rather than three nested folders holding one file. An article’s date is taken from the most reliable signal on the page, in this order: meta tags, JSON-LD, the URL, a `<time>` element and finally the HTTP headers.

The tests cover date extraction, URL handling, link rewriting, the cache, sitemap and feed parsing, navigation parsing and page classification. They’re pure logic with no network calls, and run in a few seconds.

## How to use it

```bash
pip install .
scrapling install
scrapepy https://docs.example.com/api/ --stay-in-path

```

The HTML copy goes to `ScrapedSites/` and the Markdown to `MarkdownSites/`.

Use it responsibly. Follow each site’s terms of service, respect its `robots.txt` and rate limits, and only keep content you have the right to keep.

---

Index of this site: [llms.txt](https://www.dealmobile.co.uk/llms.txt)