Working system
ILCrawler
Docker-first technical SEO crawler for authorized audits, combining raw crawl data, rendered browser checks, Lighthouse audits, issue generation, CSV exports, and a private review UI.
Stack
FastAPI, PostgreSQL, MinIO/S3-compatible storage, Docker Compose, raw HTTP crawler, Playwright rendered audits, local Lighthouse audits, worker processes, bearer-token API auth, CSV exports.
What it does
- runs raw HTTP crawls with robots.txt, sitemap discovery, URL normalization, and crawl limits
- captures metadata, canonicals, headings, robots directives, hreflang, depth, word count, TTFB, and link graphs
- records internal and external links for review and export
- generates issue rows for duplicate metadata/content, broken links, orphan/dead-end pages, canonical problems, redirects, and hreflang issues
- runs Playwright rendered audits and local Lighthouse checks
- exports pages, issues, internal links, and external links as CSV
- shows crawl status, issue summaries, pages, links, rendered audits, and Lighthouse output in a private dashboard
Scoped for owned and client-authorized audits.
Run summary
Audit output
Pages crawled
500
Links found
72,340
Issue rows
12,434
UI screenshots
ILCrawler in use
Private UI excerpts from a completed crawl run. Tokens and sensitive identifiers are removed.