Open-source technical SEO CI gate

crawlcontract — deterministic crawl and indexability CI

An open-source Rust CLI that turns static output or a live site into one URL-state graph, reports deterministic findings when crawl and indexability signals contradict each other, and fails CI according to the configured severity gate.

Record: crawlcontract
Release: Version 0.3.1
License: MIT OR Apache-2.0
Language: Rust

RustTokioHTTPXML sitemapsSARIFGitHub Actions

Inspect source View v0.3.1 release

Engineering record

crawlcontract

The current public release is v0.3.1. The source and release artifacts make the implementation inspectable, but the project is in early development and snapshot compatibility is not promised before the first stable release.

Indexing and crawl fixes Crawler implementation system

Verified behavior

System behavior and guardrails

Builds one URL-state graph from static files or live HTTP evidence
Reports cross-artifact contradictions with deterministic rule IDs and structured evidence
Compares baseline and candidate snapshots with configurable loss gates
Emits terminal, JSON, Markdown, and SARIF reports with stable CI exit codes

Implementation log

Design, operation, and verification notes

What the contract checks

Sitemap URLs resolve, remain crawlable and indexable, and identify the expected canonical
HTML and HTTP canonical signals agree; canonical targets resolve without chains or cycles
Meta robots and X-Robots-Tag directives combine restrictively with robots.txt crawl rules
Internal links resolve directly to verified targets and indexable pages remain reachable from the entry point
Hreflang alternates are reciprocal, self-inclusive, and aligned with canonical targets

CI and release workflow

Scans a static build directory or a bounded-concurrency live or preview URL
Maps a preview origin onto the intended public URL identity with --public-origin
Applies policy exclusions, failure severities, and snapshot-diff loss thresholds
Writes terminal, JSON, Markdown, or SARIF findings and returns CI-specific exit codes
Publishes checksummed binaries for Linux, macOS, and Windows from tagged releases

Current boundaries

The v0.3.1 release uses snapshot schema 1.0, and compatibility is not promised before the first stable release. Live scans inspect response HTML rather than a browser-rendered DOM, sitemap discovery currently follows one index level without .xml.gz support, and offline scans cannot verify cross-origin targets.

Search IndexLane