io.github.nolindnaidoo/scrape-le is an MCP server that analyse robots.txt content and report whether a path may be crawled. Its tool list has not been published yet over stdio, sse and http, requires no API key, and scores 59/100 on MCPpedia's security, maintenance and efficiency rubric.
Config is the same across clients — only the file and path differ.
{
"mcpServers": {
"scrape-le": {
"args": [
"-y",
"scrape-le-mcp"
],
"command": "npx"
}
}
}Are you the author?
Add this badge to your README to show your security score and help users find safe servers.
Useful? A star or rating is how other developers find it — > ★ GitHub · > ★ Open VSX · > ★ Marketplace
Run this in your terminal to verify the server starts. Then let us know if it worked — your result helps other developers.
npx -y 'scrape-le-mcp' 2>&1 | head -1 && echo "✓ Server started successfully"
After testing, let us know if it worked:
Five weighted categories — click any category to see the underlying evidence.
No known CVEs.
Checked scrape-le-mcp against OSV.dev.
Be the first to review
Have you used this server?
Share your experience — it helps other developers decide.
Sign in to write a review.
Others in browser
Monitor browser logs directly from Cursor and other MCP compatible IDEs.
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
Self-hosted URL- and file-to-Markdown service for humans and AI agents - web pages, documents, images, audio, YouTube. PWA + REST + MCP + Claude Code skill, Reddit-aware, refreshable share links.
MCP Security Weekly
Get CVE alerts and security updates for io.github.nolindnaidoo/scrape-le and similar servers.
Start a conversation
Ask a question, share a tip, or report an issue.
Sign in to join the discussion.
Load a URL in headless Chromium and see what will block your scraper — before you write it
Anti-bot vendors, rate limits, robots.txt rules, login walls, console errors, screenshots
Useful? A star or rating is how other developers find it — ★ GitHub · ★ Open VSX · ★ Marketplace
Run Scrape-LE: Check URL Scrapeability (Ctrl+Alt+S / Cmd+Alt+S), enter a URL, and the page loads in a real headless Chromium. The report lands in the output channel: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections. Works in VS Code and VS Code–based editors like Cursor and VSCodium (installable from Open VSX).
One-time setup: run Scrape-LE: Setup Browser to install Chromium (~130MB, into Playwright's browser cache).
| Where | What you get | Install |
|---|---|---|
| VS Code | The same check, in your editor, on a keystroke | Marketplace |
| Cursor, VSCodium, Windsurf | The same extension | Open VSX |
| A terminal or a CI step | The same run over a whole tree, with exit codes | cargo install scrape-le · crates.io |
| Any MCP agent, via Node | analyze_robots_txt over stdio | npx scrape-le-mcp · npm |
| Zed | The MCP server as a context server | add it by hand (no listing yet) |
The same engine runs as an MCP server, so an agent can call it directly instead of you running a command.
| Editor | How |
|---|---|
| VS Code 1.101+ | Nothing to install — the extension registers analyze_robots_txt with agent mode |
| Zed | No listing yet — add the MCP server by hand |
| Claude Code | claude mcp add scrape-le -- npx -y scrape-le-mcp |
| Cursor, Windsurf, anything else | point it at npx scrape-le-mcp |
analyze_robots_txt(content, path, maxResults?)
Given robots.txt contents and a path, reports whether the generic (User-agent: *) rules permit crawling it, plus the crawl delay, disallowed patterns and any sitemaps.
The server takes con