TheCrawler — AI-ready web scraper with validated extraction contracts

Scrape web pages, run LLM-powered structured extraction, or diagnose whether URLs are ready for a built-in extraction contract before spending LLM tokens. Open source engine (AGPL-3.0). $0.005 per successfully scraped page on Apify.

Start with a safe test: run one public URL with dryRun: true on Apify, or clone the current GitHub source and run the local CLI/MCP build from engine/. A small proof pack is in examples/diagnostic-challenge, including a sample readiness report at examples/diagnostic-challenge/sample-report.md.

$500 extraction readiness sprint

Use this when you need to know whether one real public-web workflow is worth automating before you spend engineering time on extraction.

• Scope: up to 25 public URLs and one target output shape.
• First step: send a public fit check through the structured issue form or use the private fit-check path on the sprint page.
• Payment: requested only after the workflow looks like a fit, by one-off $500 payment link or invoice.
• Output: a readiness report with ready, mixed, blocked, or not-worth-automating-yet guidance.
• Credit: if the workflow continues into setup or hosted usage, the $500 is credited toward that next step.

The public offer thread is GitHub issue #1. The proof pack includes a sample readiness report showing the report shape before a buyer sends URLs.

Public fit checks should use this shape:

Workflow type:
Public URLs (up to 25):
Target output shape / required fields:
Known blockers or constraints:
Timing:

Do not include login credentials, private URLs, personal data, or raw customer data in GitHub issues.

What makes this different

• Validated extraction contracts: select a built-in contract, get normalized data plus validation.valid, required fields, and missing-field evidence. Current contracts: real-estate-listing, product-page, docs-page.
• Brand identity extraction (extractBrand: true): one call returns the site's ranked color palette, themeColor, and best-guess logo candidates (JSON-LD / header SVG / favicons / og:image). In Playwright mode it reads rendered colors via getComputedStyle — works on SPAs where static CSS can't. Deterministic, no LLM.
• Content controls: onlyMainContent plus includeTags / excludeTags (CSS allow/deny) strip nav, footer, sidebars, and ads from text, markdown, links, and HTML output. Firecrawl-compatible. waitFor alias supported.
• HTML formats: extractHtml (cleaned, main-content HTML) and extractRawHtml (full serialized DOM) alongside markdown.
• No-LLM diagnostics: run diagnoseMode to score source readiness, identify blockers, and save a buyer-readable Markdown report before extraction.
• LLM-powered extraction: send a JSON Schema or use a contract, get parsed typed data back. Endpoint-agnostic — point at OpenAI, your own llama.cpp / vLLM / LM Studio / Ollama. You bring the LLM, no vendor lock-in.
• Adaptive crawling: Cheerio first (fast HTTP+parse), auto-fall-back to Playwright when an SPA shell is detected. Keeps browser rendering optional instead of mandatory for every page.
• Structured errors: errorType enum (dns | timeout | rate-limit | blocked-bot | js-required | http-4xx | http-5xx | parse | network | unknown) + errorRetryable boolean. Agents branch programmatically — no regex on error strings.
• Challenge-page detection: 200 OK responses with access-control or challenge-page bodies are flagged as errorType: 'blocked-bot' instead of returning challenge HTML as useful content.
• **Out-of-box extractors

... View full README on GitHub