# Web Crawler

> A small, dependency-minimal Rust web crawler that fetches a seed URL, extracts same-host links from the homepage, and saves HTML responses to disk.
Published: 2026-03-24
Tags: rust, web-crawler, scraping, tokio, cli

Source: https://hugo-portfolio-teal.vercel.app/projects/web-crawler/

<!-- synced from GitHub README (scripts/sync-readmes.mjs) — edit the repo README, not this file -->

# crawler

A small, dependency-minimal Rust web crawler that fetches a seed URL, extracts same-host links from the homepage, and saves HTML responses to disk.

## 🔍 What it does
- Accepts a seed URL (or hostname) as CLI input
- Fetches the homepage once
- Extracts `<a href="...">` links in the first page
- Normalizes each link to an absolute URL
- Follows only same-host links
- Fetches each same-host page once
- Saves each response body in `out_dir` using a deterministic URL hash filename
- Logs crawl events to stdout with status and byte counts

## 🧩 Project structure
- `src/main.rs` - CLI entrypoint using `clap` + `tokio`
- `src/lib.rs` - exposes reusable crawler API
- `src/engine.rs` - crawl orchestration
- `src/fetch.rs` - HTTP fetch wrapper with `reqwest`
- `src/links.rs` - HTML link extraction
- `src/storage.rs` - file path generation, save HTML
- `src/url_util.rs` - URL normalization and same-host checks
- `src/log.rs` - logging abstraction (stdout + pluggable)

## ▶️ Usage
Build and run from project root:

```bash
cargo run --release -- "https://example.com" --out-dir crawl_out
```

Short form:

```bash
cargo run --release -- example.com -o crawl_out
```

Defaults:
- `out_dir`: `crawl_out`

## 🚦 Output
- `crawl_out/<url_hash>.html`
- `url_hash` is derived from normalized final URL
- stdout log events include: `seed`, `response`, `fetch`, `save`, `skip_links`, `link_skip`, `fetch_err`, `save_err`

## 🛠️ Configuration
No configuration file. Use CLI args only.

## 🧪 Tests
No test files are currently included. The library is unit-test-friendly via `Crawler::with_logger` and `CrawlConfig`.

## 📦 Dependencies
- `reqwest` (HTTP client)
- `tokio` (async runtime)
- `anyhow` (error handling)
- `clap` (CLI)
- `url` (URL parsing)

## 💡 Extending
- add depth control (breadth-first / recursive crawl)
- add robots.txt + rate limiting
- add concurrency queue and dedupe URL set
- add filter rules (patterns, content types)
- instrument with structured logging / metrics

## 📝 Notes
The crawler is intentionally simple, for learning and small local crawl tasks. It is not a production spider and does not enforce politeness controls by default.