Rewriting an old PHP scraper around curl_multi

Scrape downloads files from web servers that have directory listings turned on, the “Index of /” pages a server shows for a folder with no home page. It’s my oldest public repository. The first commit is from November 2013, it became a class in May 2014, and in October 2025 its download loop was rewritten around cURL’s multi interface. The repository is archived now, so the class stays as it is. This is what the rewrite changed, and what turned up when I pointed the current class at a listing laid out like Apache’s.

One file at a time

The version before the rewrite handled each file in turn. It asked the server for the file’s headers with get_headers(), following any Location header, to learn its length. Then it downloaded with a Range header into a temporary file in the cache folder, and appended that to the real file, looping until the file on disk was as long as the server said. If a transfer stopped short, the next pass of the loop asked for the rest.

That was careful, and slow: one file at a time, with an extra request for the headers before each download started.

A pool of transfers

The rewrite keeps a queue of files and a pool of active transfers, ten by default and at most 50. This is the core of downloadInParallel() in scraper.class.php:

while (!empty($queue) || !empty($active)) {
    // Fill the active pool
    while (count($active) < $this->maxConcurrentDownloads && !empty($queue)) {
        $link = array_shift($queue);
        // ...skip long paths, create the folder, skip files already on disk...
        $ch = $this->createCurlHandle($link['link'], $link['save_path']);
        curl_multi_add_handle($mh, $ch);
        // ...remember the handle in $active, keyed by (int)$ch...
    }

    // Execute the multi-handle
    if (!empty($active)) {
        do {
            $status = curl_multi_exec($mh, $running);
        } while ($status === CURLM_CALL_MULTI_PERFORM);

        // Wait for activity
        if ($running > 0) {
            curl_multi_select($mh, 0.5);
        }
        // ...collect finished transfers with curl_multi_info_read()...
    }
}

Each pass tops the pool up, lets cURL move every transfer along, and waits up to half a second for one of them to have something to do. Then it reads off the finished ones. A transfer counts as a success only if cURL reports no error and the status is 200. Finished handles are removed, which frees their places for the next pass.

Each handle writes straight to its file. createCurlHandle() opens the destination and passes it to cURL with CURLOPT_FILE, so a file streams to disk as it arrives and memory use doesn’t grow with its size. The handles also share a cache:

// Connection pooling for better performance
if ($this->curlShareHandle === null) {
    $this->curlShareHandle = curl_share_init();
    curl_share_setopt($this->curlShareHandle, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);
    curl_share_setopt($this->curlShareHandle, CURLSHOPT_SHARE, CURL_LOCK_DATA_SSL_SESSION);
    curl_share_setopt($this->curlShareHandle, CURLSHOPT_SHARE, CURL_LOCK_DATA_CONNECT);
}
curl_setopt($ch, CURLOPT_SHARE, $this->curlShareHandle);

DNS answers, TLS sessions and open connections are shared between handles, so many files from one server don’t each need a fresh lookup and handshake. A transfer that runs slower than 10KB a second for a minute is abandoned, and there are connection and transfer timeouts, 30 and 300 seconds by default, and an optional bandwidth limit.

Reading the listings

The downloads run in parallel, but the listing pages are still fetched one at a time as the class walks the tree. The rewrite also replaced the way it reads them. Links come out of each page with a regular expression, and this is how they’re checked and made absolute:

// Skip parent directory and same directory
if ($href === '.' || $href === '..' || $href === '../') {
    continue;
}

// Make absolute URL
if (strpos($href, 'http') !== 0) {
    $href = rtrim($var['scrape_url'], '/') . '/' . ltrim($href, '/');
}

A link that ends in a slash is a folder to walk into, and anything else is a file. That works for listings that use relative links, such as nginx’s, which link to the parent as ../.

Apache’s listings have two kinds of link this doesn’t expect. The column headings link to ?C=N;O=D and similar, to sort the table. And the Parent Directory link is an absolute path, such as /files/, which this code glues onto the end of the current URL.

The old version handled both. It parsed each page with DOMDocument, joined absolute paths to the host, resolved .., skipped any link shorter than the page it was on, and skipped any link with a ? in it.

What a test listing showed

I served a small tree with PHP’s built-in web server and a router that writes listings in Apache’s style. It has a files/ folder holding report.pdf, cat.gif and a sub/ folder with notes.pdf in it, and each listing also links to a missing.pdf that returns a 404.

With a filter for PDFs, the class downloaded everything twice. On /files/, the Parent Directory link is /, and the join turns that back into /files/, which looks like a folder called files to walk into. So every file arrived under example/ and again under example/files/. On /files/sub/, the parent link became /files/sub/files/, which cost a request and a 404.

With no filter, the sort links became files too. Each folder got copies of its own listing named C N O D, C M O A and C S O A, after the file name cleaner had turned the punctuation into spaces. Three real files became 22 downloads.

The 404s were the other surprise. Each missing.pdf was saved, holding the server’s 44-byte “Not Found” page, and counted as failed. On the next run, the class found 0 files to process. It skips any file that already exists on disk, so a 404 page saved under a file’s name counts as done, and so would a file that was cut off half way. The comment above that check says “Check if file already exists with correct size”, but the size isn’t checked.

Settings that do nothing

The README written with the rewrite promises automatic retries with exponential backoff. setMaxRetries() and setRetryDelay() store their values, and nothing reads them. setCachePath() is left over from the old version, which kept partial downloads in the cache folder. The rewrite writes straight to the destination, so that setting does nothing either. Calling them is harmless, which keeps older code working, but they don’t change anything.

Using it

The repository is archived, so this is a class to read or copy rather than install. Copy scraper.class.php into a project, and run a search first, which lists what would be downloaded without creating anything:

require_once 'scraper.class.php';

$scraper = new scraper();
$scraper->setDestinationRoot('./downloads/')
    ->setMode('search')
    ->addLocation('https://example.com/files/', 'example')
    ->excludeInPath('?')
    ->scrape();

With no type filter, excludeInPath('?') keeps Apache’s sort links out, as it did on my test tree. Nothing in the class prevents the doubled tree, but a search shows it before anything is downloaded. When the list looks right, remove setMode('search') and run it again from the command line:

php scrape.php

scrape() turns off PHP’s time limit and prints its progress as it goes. Only download files you have permission to download. The code is on GitHub.