Scrape

Scrape is a PHP class for one job: downloading files from a web server that has directory listings switched on, the plain “Index of /” pages a server shows for a folder with no home page. It’s my oldest public repository. I started it in November 2013 and added to it on and off for years.

What it does

You give it one or more starting URLs, each with the local folder to save into and, optionally, the file types you want. It follows the listing into every subfolder and downloads the files that match.

Filter by file extension, and leave out paths or file names that contain given words.
Keep only files whose names contain search terms.
Search mode lists the matching files without downloading them, and test mode creates the folder structure with empty files.
Download a random sample of a given number of files instead of all of them.
Rename files before they’re saved, with your own function. The demo strips a 14-digit date stamp from the start of each name.
Skip files that are already on disk, so a stopped run can be started again.

How it’s built

It’s a single PHP class with a chainable set of setters, and it needs only PHP and the cURL extension.

The first version, in November 2013, was a single script: you set a few constants and a list of URLs, and ran it from the command line. In May 2014 it became a class. It read each listing page with cURL, pulled the links out of the HTML and downloaded one file at a time, creating local folders only when it had a file to write into them. Later in 2014 came the search option, the file name processor and random selection.

In 2025 the download loop was rewritten around cURL’s multi interface, so it downloads several files at once, ten by default. It shares DNS, connection and SSL session caches between requests. Each file streams straight to disk, so memory use stays flat however large the files are. It also gained connection and transfer timeouts, an optional bandwidth limit, a log file, a progress callback and statistics for the run.

How to use it

Copy scraper.class.php into your project:

require_once 'scraper.class.php';

$scraper = new scraper();
$scraper->setDestinationRoot('./downloads/')
    ->setMaxConcurrentDownloads(10)
    ->addLocation('https://example.com/files/', 'example', ['pdf'])
    ->excludeInFilename(['draft', 'backup'])
    ->scrape();

Only download files you have permission to download.