Private website audit tool

ILCrawler — the website audit tool I use

I built ILCrawler to keep page checks, links, content comparisons, performance samples, and audit findings together. I use it on my own websites and client sites where I have permission to run an audit.

Project: ILCrawler

GoRustPythonNode.jsNext.jsPostgreSQLS3-compatible storage

About the project

ILCrawler

Client data is kept private.

Property listing indexing and headless QA

What it does

Discovers pages from robots.txt, sitemaps, and internal links while enforcing project scope and crawl limits
Keeps HTTP checks, browser-rendered pages, and Lighthouse or PageSpeed samples separate
Groups issues by type, severity, confidence, review status, and affected pages
Retains run progress, crawl errors, links, resources, content comparisons, screenshots, and exportable handoff data

How it works

What an audit records

Response state, indexability signals, metadata, canonicals, headings, robots directives, hreflang, media counts, depth, word count, and timing
Internal and external links, plus images, scripts, stylesheets, fonts, iframes, and other page resources
Browser-rendered pages and performance results for the selected samples
Issue history, affected-page examples, content similarity, and review status

How I use it

Choose the site section to crawl and which rendering or performance checks to run
Follow the crawl and extra checks in one audit view
Review grouped problems and open the affected pages
Export a client summary or a detailed technical report

What the screenshots show

These screenshots show version 0.32.1, using IndexLane or client records with identities redacted. They show the tool interface, not search or business results.

ILCrawler is private software for my own and client-authorized audits. It is not a public URL checker, and the version shown does not include Search Console, analytics, or third-party SEO data integrations.

Screenshots

ILCrawler v0.32.1 IndexLane audit setup with raw crawl, JavaScript rendering, PageSpeed, and full-audit presets. — Choose the pages to crawl and optional rendering or PageSpeed checks. Open full-resolution image.

ILCrawler v0.32.1 IndexLane run detail with crawl, render, performance, similarity, issue, asset, and review state. — An audit with crawl progress, content comparisons, grouped issues, and page resources. Open full-resolution image.

ILCrawler v0.32.1 issue groups with explicit sampling limits and severity, state, and site filters. — Filter audit issues by website, severity, and review status. Open full-resolution image.

ILCrawler v0.32.1 site inventory with client identities redacted and audit state, issue count, scope, and robots handling visible. — Website audit status and issue counts. Client identities are redacted. Open full-resolution image.

Search IndexLane