Web Scraping With PHP – Tools and Patterns
Learn how to scrape in PHP: Guzzle for fetching, DomCrawler for parsing, concurrent requests with promises, and handling rendered pages.
PHP has capable collection libraries, and it is often the pragmatic choice when the surrounding application is already PHP — the collected data usually needs to land there anyway.
1. Fetching With Guzzle
$client = new GuzzleHttp\Client([
'timeout' => 30,
'headers' => ['User-Agent' => 'example-collector/1.0'],
]);
$response = $client->request('GET', $url);
if ($response->getStatusCode() !== 200) {
throw new RuntimeException('Unexpected status ' . $response->getStatusCode());
}
$html = (string) $response->getBody();
Guzzle handles connection reuse, redirects, cookies, and proxies, and it throws on 4xx and 5xx by default — which is safer than a client that returns an error page silently.
2. Parsing With DomCrawler
use Symfony\Component\DomCrawler\Crawler;
$crawler = new Crawler($html, $url);
$crawler->filter('article.product')->each(function (Crawler $node) {
$title = trim($node->filter('h3')->text(''));
$price = trim($node->filter('.price')->text(''));
});
DomCrawler supports CSS selectors and XPath, and passing a default to text() avoids exceptions on missing elements — the most common source of crashes in PHP extraction code.
3. Concurrency With Promises
Guzzle's asynchronous requests plus a pool bound concurrency:
$pool = new GuzzleHttp\Pool($client, $requests, [
'concurrency' => 8,
'fulfilled' => function ($response, $index) { /* handle */ },
'rejected' => function ($reason, $index) { /* log */ },
]);
$pool->promise()->wait();
The concurrency setting is the limiter, and eight to sixteen against a single host is usually plenty.
4. Encoding
PHP's string handling is byte-oriented, so encoding must be explicit. Convert pages to UTF-8 on the way in using the declared charset, and use the multibyte string functions for anything involving character counts or case changes.
Skipping this produces the familiar garbled output on non-English pages.
5. Long-Running Scripts
Collection scripts run far longer than a web request, so the defaults work against them: execution time limits, memory limits, and output buffering all need attention when running from the command line.
Streaming output to a file as records are extracted, rather than accumulating an array, keeps memory flat regardless of how many pages are collected.
6. Rendered Pages
PHP does not execute JavaScript. Options are calling the page's own JSON endpoint, or driving a browser — Symfony Panther provides a familiar interface over WebDriver for that purpose.
As always, checking for the endpoint first is cheaper than rendering.
7. Practical Notes
- Use Composer and pin versions.
- Set timeouts on every request.
- Provide defaults when reading optional nodes.
- Write JSON Lines rather than building large arrays.
- Bound concurrency per host and pace requests.
8. Summary
PHP collects effectively with Guzzle for fetching, DomCrawler for parsing, and a request pool for bounded concurrency, with Panther available when rendering is required.
Handle encoding explicitly, give selectors defaults, stream output rather than accumulating it, and look for the underlying JSON endpoint before adding a browser.