crawler

#

A small, dependency-minimal Rust web crawler that fetches a seed URL, extracts same-host links from the homepage, and saves HTML responses to disk.

🔍 What it does

#

• Accepts a seed URL (or hostname) as CLI input
• Fetches the homepage once
• Extracts <a href="..."> links in the first page
• Normalizes each link to an absolute URL
• Follows only same-host links
• Fetches each same-host page once
• Saves each response body in out_dir using a deterministic URL hash filename
• Logs crawl events to stdout with status and byte counts

🧩 Project structure

#

• src/main.rs - CLI entrypoint using clap + tokio
• src/lib.rs - exposes reusable crawler API
• src/engine.rs - crawl orchestration
• src/fetch.rs - HTTP fetch wrapper with reqwest
• src/links.rs - HTML link extraction
• src/storage.rs - file path generation, save HTML
• src/url_util.rs - URL normalization and same-host checks
• src/log.rs - logging abstraction (stdout + pluggable)

▶️ Usage

#

Build and run from project root:

cargo run --release -- "https://example.com" --out-dir crawl_out

Short form:

cargo run --release -- example.com -o crawl_out

Defaults:

• out_dir: crawl_out

🚦 Output

#

• crawl_out/<url_hash>.html
• url_hash is derived from normalized final URL
• stdout log events include: seed, response, fetch, save, skip_links, link_skip, fetch_err, save_err

🛠️ Configuration

#

No configuration file. Use CLI args only.

🧪 Tests

#

No test files are currently included. The library is unit-test-friendly via Crawler::with_logger and CrawlConfig.

📦 Dependencies

#

• reqwest (HTTP client)
• tokio (async runtime)
• anyhow (error handling)
• clap (CLI)
• url (URL parsing)

💡 Extending

#

• add depth control (breadth-first / recursive crawl)
• add robots.txt + rate limiting
• add concurrency queue and dedupe URL set
• add filter rules (patterns, content types)
• instrument with structured logging / metrics

📝 Notes

#

The crawler is intentionally simple, for learning and small local crawl tasks. It is not a production spider and does not enforce politeness controls by default.