Web Scraping With Java – Tools and Patterns
Learn how to scrape in Java: the built-in HTTP client, jsoup for parsing, concurrency with executors, and where Selenium fits.
Java has a modern HTTP client in the standard library, a mature HTML parser in jsoup, and strong concurrency primitives — enough for production collection without third-party HTTP dependencies.
1. Fetching
HttpClient client = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
HttpRequest request = HttpRequest.newBuilder(URI.create(url))
.header("User-Agent", "example-collector/1.0")
.timeout(Duration.ofSeconds(30))
.GET()
.build();
HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
if (response.statusCode() != 200) {
throw new IOException("Unexpected status " + response.statusCode());
}
The client supports HTTP/2 by default and offers an asynchronous variant returning a CompletableFuture.
2. Parsing With jsoup
Document doc = Jsoup.parse(response.body(), url);
for (Element card : doc.select("article.product")) {
String title = card.selectFirst("h3").text().trim();
String price = card.selectFirst(".price").text().trim();
}
jsoup is tolerant of malformed markup, supports CSS selectors, and can fetch pages itself — though using the standard client for fetching gives finer control over timeouts and headers.
3. Concurrency
An executor with a bounded pool is the straightforward approach; the asynchronous client with a semaphore works equally well. Either way the limit should be per host rather than only global.
Virtual threads change the calculus for blocking I/O, making a thread-per-request model practical at scale where it previously was not.
4. Structured Output
Records map cleanly to extracted data, and a JSON library serialises them to JSON Lines for streaming output:
record Product(String url, String title, BigDecimal price, Instant collectedAt) {}
Using BigDecimal for money rather than a floating-point type avoids a class of rounding error that is difficult to detect afterwards.
5. When a Browser Is Needed
jsoup parses; it does not execute JavaScript. For client-rendered pages, Selenium drives a real browser through the standardised WebDriver protocol, with mature Java bindings.
Before reaching for it, check whether the page calls a JSON endpoint that can be requested directly.
6. Practical Notes
- Set both connect and request timeouts.
- Check status codes explicitly; the client does not throw on 4xx or 5xx.
- Reuse one client, which holds the connection pool.
- Handle encoding using the declared charset rather than the platform default.
- Bound concurrency per host and pace requests.
7. Where Java Fits
It suits collection embedded in an existing Java system, work that must integrate with enterprise infrastructure, and teams whose operational tooling is already built around the JVM.
For a standalone script, lighter ecosystems get there faster; for a service that must run alongside existing Java components, this is the path of least friction.
8. Summary
Java collects with the standard HTTP client, jsoup for parsing, and executors or virtual threads for concurrency, with Selenium for pages that require rendering.
Set explicit timeouts, check statuses, reuse the client, use decimal types for money, and look for the underlying endpoint before adding a browser.