ScrapePy

ScrapePy is a command-line tool for keeping a copy of a website: documentation you want to read on a train, a reference site you want to search locally, or articles you want to feed into your own tools as Markdown. You give it a URL. It crawls the site, saves every page with its images, and rewrites the links so the copy works in a browser with no connection.

What it does

• Crawls every page on the same domain, or only pages under a starting path.
• Saves the HTML and the images, and by default the CSS, JavaScript and fonts too. It rewrites internal links to relative paths, so pages link to each other on disk.
• Writes a Markdown copy of the site alongside, with YAML frontmatter on every file: title, URL, date, page type, word count and description. Images are copied across so the Markdown links work.
• Re-runs incrementally. It sends the ETag and Last-Modified values from the last crawl, skips pages the server says haven’t changed, and compares a SHA-256 hash of the content for servers that don’t send those headers.
• Records broken links in a text file.
• Reads robots.txt, sitemaps and RSS or Atom feeds before crawling, and parses the home page’s navigation, including dropdown menus, to map the site’s sections first.
• Can keep only recent articles on news sites, for example the last two weeks, while keeping every section page.

It also has modes for single-page apps, documentation platforms and Swagger or OpenAPI references. An audit mode looks at a site’s structure without crawling it and saves test fixtures.

How it’s built

ScrapePy needs Python 3.11 or later. Fetching uses Scrapling: plain HTTP requests for ordinary pages, and a real Chromium browser for pages that need JavaScript to render. Parsing uses BeautifulSoup and lxml, asset downloads run concurrently with httpx, and the Markdown comes from markdownify with trafilatura as a fallback for pulling out the main content.

A few details matter for offline copies. URLs are mapped to files carefully: /about becomes about.html, and a path that tries to climb out of the output folder falls back to a flat file name instead. Numeric path segments are folded into the file name, so /news/12345/1/slug becomes news/slug-12345-1.html rather than three nested folders holding one file. An article’s date is taken from the most reliable signal on the page, in this order: meta tags, JSON-LD, the URL, a <time> element and finally the HTTP headers.

The tests cover date extraction, URL handling, link rewriting, the cache, sitemap and feed parsing, navigation parsing and page classification. They’re pure logic with no network calls, and run in a few seconds.

How to use it

pip install .
scrapling install
scrapepy https://docs.example.com/api/ --stay-in-path

The HTML copy goes to ScrapedSites/ and the Markdown to MarkdownSites/.

Use it responsibly. Follow each site’s terms of service, respect its robots.txt and rate limits, and only keep content you have the right to keep.