MarkGrab

한국어 문서 · llms.txt

Universal web content extraction — any URL to LLM-ready markdown.

from markgrab import extract

result = await extract("https://example.com/article")
print(result.markdown)    # clean markdown
print(result.title)       # "Article Title"
print(result.word_count)  # 1234
print(result.language)    # "en"

Features

• HTML — BeautifulSoup + content density filtering (removes nav, sidebar, ads)
• YouTube — transcript extraction with timestamps
• PDF — text extraction with page structure
• DOCX — paragraph and heading extraction
• Auto-fallback — tries lightweight httpx first, falls back to Playwright for JS-heavy pages
• Async-first — built on httpx and Playwright async APIs

Install

pip install markgrab

Optional extras for specific content types:

pip install "markgrab[browser]"    # Playwright for JS-rendered pages
pip install "markgrab[youtube]"    # YouTube transcript extraction
pip install "markgrab[pdf]"       # PDF text extraction
pip install "markgrab[docx]"      # DOCX text extraction
pip install "markgrab[all]"       # everything

Usage

Python API

import asyncio
from markgrab import extract

async def main():
    # HTML (auto-detects content type)
    result = await extract("https://example.com/article")

    # YouTube transcript
    result = await extract("https://youtube.com/watch?v=dQw4w9WgXcQ")

    # PDF
    result = await extract("https://arxiv.org/pdf/1706.03762")

    # Options
    result = await extract(
        "https://example.com",
        max_chars=30_000,       # limit output length (default: 50K)
        use_browser=True,       # force Playwright rendering
        stealth=True,           # anti-bot stealth scripts (opt-in)
        timeout=60.0,           # request timeout in seconds
        proxy="http://proxy:8080",
    )

asyncio.run(main())

CLI

markgrab https://example.com                     # markdown output
markgrab https://example.com -f text             # plain text
markgrab https://example.com -f json             # structured JSON
markgrab https://example.com --browser           # force browser rendering
markgrab https://example.com --max-chars 10000   # limit output

ExtractResult

result.title        # page title
result.text         # plain text
result.markdown     # LLM-ready markdown
result.word_count   # word count
result.language     # detected language ("en", "ko", ...)
result.content_type # "article", "video", "pdf", "docx"
result.source_url   # final URL (after redirects)
result.metadata     # extra metadata (video_id, page_count, etc.)

How it works

flowchart TD
    A["🔗 URL Input"] --> B{"Content\nType?"}
    B -->|"HTML"| C["HTTP fetch\n(httpx)"]
    C --> D{"JS\nrequired?"}
    D -->|"no"| E["HTML Parser\n→ clean markdown"]
    D -->|"yes"| F["Playwright\nfallback"]
    F --> E
    B -->|"YouTube"| G["Transcript API\n→ timestamped markdown"]
    B -->|"PDF"| H["PDF Parser\n→ structured markdown"]
    B -->|"DOCX"| I["DOCX Parser\n→ markdown"]
    E --> J["✅ LLM-ready\nMarkdown"]
    G --> J
    H --> J
    I --> J

For HTML pages, if the initial httpx fetch yields fewer than 50 words, MarkGrab automatically retries with Playwright to handle JavaScript-rendered content.

Disclaimer

This software is provided for legitimate purposes only. By using MarkGrab, you agree to the following:

• robots.txt: MarkGrab does not check or enforce robots.txt. Users are solely responsible for check

... View full README on GitHub