Fetching a feed from a URL you don’t trust

RSS to JSON reads a feed from a URL and gives back an array ready for json_encode(). That’s harmless when I choose the URL. It’s a different matter when a visitor types it into a form, because the request comes from the server, and the server can reach places the visitor can’t: an admin page on 127.0.0.1, a service on the private network, or 169.254.169.254, where cloud servers read their own settings and, on some setups, their credentials. Getting a server to make requests like that on your behalf is called server-side request forgery.

Version 2.1 added an option for it, block_private_ips. It’s off by default, so existing code behaves as before, and it’s meant for any URL that comes from someone you don’t trust. Here’s what it does, in the order it does it.

Which addresses count as private

The list lives in src/IpAddress.php. I wrote the ranges out rather than rely on PHP’s FILTER_FLAG_NO_PRIV_RANGE and FILTER_FLAG_NO_RES_RANGE, whose coverage has changed between PHP versions. The IPv4 list has fifteen ranges, including these:

'10.0.0.0/8',       // Private
'100.64.0.0/10',    // Carrier-grade NAT (includes 100.100.100.200, Alibaba Cloud metadata)
'127.0.0.0/8',      // Loopback
'169.254.0.0/16',   // Link-local (includes 169.254.169.254, AWS/GCP/Azure metadata)
'172.16.0.0/12',    // Private
'192.168.0.0/16',   // Private

The others are the “this network” block, the documentation and benchmarking ranges, the old 6to4 relay range, multicast, and the reserved block that ends at 255.255.255.255. IPv6 has its own list: the unspecified and loopback addresses, unique local addresses (where AWS puts its IPv6 metadata address, fd00:ec2::254), link-local, site-local, multicast, documentation, discard and Teredo.

An IPv6 address can also carry an IPv4 address inside it. ::ffff:169.254.169.254 is the metadata address written as IPv6, and a check that only looked at the IPv6 list would let it through. So four IPv6 ranges are marked as wrapping IPv4, and for those the embedded four bytes are checked against the IPv4 list:

private const EMBEDDED_V4 = [
    '::ffff:0:0/96' => 12,  // IPv4-mapped
    '::/96' => 12,          // IPv4-compatible (deprecated)
    '64:ff9b::/96' => 12,   // NAT64
    '2002::/16' => 2,       // 6to4
];

The number is the byte where the IPv4 address starts. 6to4 puts it straight after the first two bytes, so 2002:7f00:0001::1 is 127.0.0.1 in disguise.

Addresses are compared as bytes, after inet_pton(), and anything that function can’t parse is treated as not public. localhost, the decimal form 2130706433 and 999.1.1.1 all fail, and the tests list each one.

Resolve first, and check every address

A host name isn’t an address, so checkUrl() resolves it before anything connects: gethostbynamel() for IPv4 and dns_get_record() for IPv6. Whoever controls a domain controls its DNS, and can return one public address and one private one. So every address has to pass, not just the first:

$addresses = $isIpLiteral ? [$host] : $this->resolveHost($host);
if ($addresses === []) {
    $this->lastError = "Could not resolve host: $host";
    return null;
}
foreach ($addresses as $address) {
    if (!$this->isAddressAllowed($address)) {
        $this->lastError = sprintf('Refusing to connect to %s (%s): private or reserved address', $host, $address);
        return null;
    }
}

Before the lookup, some URLs are refused outright in this mode. A URL with a user name and password, whitespace or a backslash goes, because PHP’s parse_url() and libcurl can disagree about which part of it is the host. A URL that ends in \@169.254.169.254/ is the kind of thing that rule is for. Host names that aren’t plain ASCII, such as local%68ost, go too. A review found that libcurl decodes a percent-encoded or Unicode host name before it connects. The pin described below was keyed on the name as written, so it didn’t apply, and curl made its own lookup of the decoded name, which nothing had checked. International domains still work in their xn-- form.

Pinning the connection

Checking the address and then letting curl connect means two DNS lookups, and the answer can change between them. That’s DNS rebinding: the attacker’s DNS server answers the first lookup with a public address and a very short lifetime, and the second with 127.0.0.1. The check passes, and curl connects somewhere else.

So curl never does its own lookup. CURLOPT_RESOLVE tells it which address to use for a host and port, and it’s given the address that was checked:

if ($target['ip'] !== null) {
    // A proxy would make its own connection, out of our sight.
    curl_setopt($ch, CURLOPT_PROXY, '');

    // Connect to the address that was checked, not a fresh lookup.
    // An IP literal involves no lookup, so there is nothing to pin.
    if (filter_var($target['host'], FILTER_VALIDATE_IP) === false) {
        $address = strpos($target['ip'], ':') === false ? $target['ip'] : '[' . $target['ip'] . ']';
        curl_setopt($ch, CURLOPT_RESOLVE, [sprintf('%s:%d:%s', $target['host'], $target['port'], $address)]);
    }
}

The host name stays in the URL, so the Host header and the TLS certificate check still use it. When a host has both kinds of address, the pin prefers IPv4, which every version of libcurl accepts in CURLOPT_RESOLVE. The empty proxy setting matters too: curl picks up proxy settings from the environment, and a proxy would make its own connection, out of the check’s sight.

The same review found that an IP literal shouldn’t be pinned at all. There’s no lookup to protect, and an IPv6 literal produced a CURLOPT_RESOLVE entry that libcurl couldn’t parse, so every URL with one failed.

On PHP 8.4 and later, with libcurl 7.80 or later, there’s a second line of defence. CURLOPT_PREREQFUNCTION runs once curl has connected and before it sends the request, with the address it connected to, and the same check can abort the request there. Older versions get a check of CURLINFO_PRIMARY_IP afterwards, which throws the response away. By then the request has gone, so that one is only a backstop. The pin is what does the work.

Redirects

curl can follow redirects itself, but then the second hop would skip every check above. So CURLOPT_FOLLOWLOCATION is off, and fetchRemoteFeed() follows redirects in its own loop. Each new URL goes back through checkUrl(): the scheme, the lookup, the addresses and a fresh pin. Only then is it requested.

The new URL comes from CURLINFO_REDIRECT_URL, so a relative redirect arrives as a full URL. A feed that redirects to http://169.254.169.254/latest/meta-data/ is refused at the second hop, and so is one that redirects to http://[::ffff:169.254.169.254]/. There’s a test for each. Five redirects are allowed by default, and one deadline covers the whole fetch, so a chain of slow redirects can’t stretch the ten-second timeout.

Some of this applies with the option off. Only http and https are fetched, for the first URL and for every redirect, so a redirect to file:///etc/passwd, gopher:// or dict:// is refused either way. curl is also limited to those two protocols, as a second check.

Testing it

The remote tests run against PHP’s built-in web server on 127.0.0.1, which is exactly the kind of address the option refuses. The class has a hook for this, the protected isAddressAllowed(), and the tests use a subclass that lets 127.0.0.1 through and nothing else on loopback:

return new class (['block_private_ips' => true]) extends RSSToJson {
    protected function isAddressAllowed(string $ip): bool
    {
        return $ip === '127.0.0.1' || parent::isAddressAllowed($ip);
    }
};

So a redirect test can start at the test server and check that a hop to 10.0.0.1 or 127.0.0.2 is refused. The rebinding defence has its own test, with a subclass whose isAddressAllowed() says yes the first time and no after that, as if DNS had changed between the check and the connection. The fetch has to fail, with the connected address in the error. The suite has 112 tests, and PHPStan checks the code at its strictest level.

What it doesn’t cover

PHP’s DNS lookups can’t be interrupted, so a slow resolver can push a fetch past its timeout. And the option only covers fetching. parse() treats anything that isn’t a valid URL as a file path, and PHP’s file functions accept stream wrappers, so untrusted input belongs in parseUrl(), which refuses anything but http and https.

Using it

composer require tomgould/rsstojson

use TomGould\RSSToJson\RSSToJson;

$parser = new RSSToJson(['block_private_ips' => true]);
$feed = $parser->parseUrl($urlFromTheForm);

if ($feed === null) {
    // For example: Refusing to connect to localhost (127.0.0.1): private or reserved address
    echo $parser->getLastError(), PHP_EOL;
}

To let one internal feed server through, extend the class and override isAddressAllowed(string $ip): bool, the way the tests do. The code is on GitHub.