io.github.JakeLiuMe/webpeel

The web data layer for AI agents.
Fetch, search, crawl, extract, screenshot — one call, zero boilerplate.

Quick Start · Agent Integrations · Docs · Playground · Get API Key

The Problem

Every AI agent that touches the web rebuilds the same brittle stack: HTTP fetch → headless browser → anti-bot bypass → HTML cleanup → markdown conversion → token budgeting. Each layer fails differently. Sites change. Cloudflare rotates challenges. Your agent gets empty strings at 2 AM and your pipeline breaks.

WebPeel replaces that entire stack with one function call. It handles engine selection, anti-bot escalation, domain-specific extraction, and token optimization so your agent gets clean, structured data every time — without managing browsers, proxies, or parsing logic.

Quick Start

# Zero-install — just run it
npx webpeel "https://example.com"

# Search the web
npx webpeel search "latest AI agent frameworks"

# Crawl an entire site
npx webpeel crawl docs.example.com --max-pages 50

# Screenshot any page
npx webpeel screenshot "https://stripe.com/pricing" --full-page

# Ask a question about any page
npx webpeel ask "https://arxiv.org/abs/2401.00001" "What is the main contribution?"

Or install globally:

npm install -g webpeel

Use as a library:

import { peel } from 'webpeel';

const result = await peel('https://news.ycombinator.com');
console.log(result.markdown);   // Clean markdown, ready for your LLM
console.log(result.metadata);   // Title, tokens saved, timing, etc.

Use via API:

curl "https://api.webpeel.dev/v1/fetch?url=https://stripe.com/pricing" \
  -H "Authorization: Bearer $WEBPEEL_API_KEY"

{
  "url": "https://stripe.com/pricing",
  "markdown": "# Stripe Pricing\n\n**Integrated per-transaction fees**...",
  "metadata": {
    "title": "Pricing & Fees | Stripe",
    "tokens": 420,
    "tokensOriginal": 8200,
    "savingsPct": 94.9
  }
}

Get your free API key → · No credit card required · 500 requests/week free

Why WebPeel

🧠 55+ Domain Extractors — Not Just HTML-to-Markdown

Generic scrapers convert raw HTML to markdown and call it a day. WebPeel has purpose-built extractors for 55+ domains — Reddit, GitHub, YouTube, Amazon, ArXiv, Hacker News, Wikipedia, StackOverflow, Zillow, Polymarket, ESPN, and more. Each extractor understands the site's structure and returns clean, structured data without browser rendering.

⚡ 65–98% Token Savings

Domain extractors strip navigation, ads, sidebars, and boilerplate before content reaches you

... View full README on GitHub