md-chunk-lite

Free tool that splits Markdown into chunks by heading. Every chunk knows its heading path and the line numbers it came from. Need PDF, Word or web page files turned into chunks (with page numbers for PDFs), or whole folders? The paid doc2md does that: doc2md on Gumroad

Good for AI search (RAG), embeddings, and docs bots. Plain code, not AI, so the same file always gives the same chunks. Runs on your own computer. MIT license. Python 3.10 or newer, on Windows, Mac and Linux.

Also searched as: ES: dividir Markdown en chunks para RAG / PT: dividir Markdown em chunks / DE: Markdown in Chunks teilen / FR: découper du Markdown / JA: Markdownチャンク分割、RAG

Links

• Website — product page with prices and options
• Apify (no-install cloud version) — pay only for what you use
• Pro version on Gumroad — PDF/Word/HTML to chunks (doc2md Pro)

Before and after

Before: a Markdown file with headings, and maybe a code block or a table. After: chunks that split on headings first. Each chunk keeps its heading path and line numbers. A code block or table is not cut in the middle when it fits.

Install

pip install md-chunk-lite

Or use pipx, which keeps it in its own space:

pipx install md-chunk-lite

Want the newest code from GitHub instead?

pip install git+https://github.com/tidybytetools/md-chunk-lite

Use it

md-chunk-lite guide.md                     # writes guide.chunks.jsonl
md-chunk-lite docs/*.md -o all.jsonl       # many files into one file
md-chunk-lite guide.md -o -                # print the chunks on screen
md-chunk-lite guide.md --max-chars 800     # smaller chunks
md-chunk-lite guide.md --outline           # just show the headings
cat guide.md | md-chunk-lite -             # read from stdin

Example run:

Saved: guide.chunks.jsonl  (7 chunks, biggest 151 chars)

Each line of the .jsonl file is one chunk:

{
  "id": "guide.md#3",
  "source": "guide.md",
  "index": 3,
  "heading_path": ["Garden Guide", "Planting", "Tomatoes"],
  "heading_level": 3,
  "anchor": "tomatoes",
  "start_line": 16,
  "end_line": 24,
  "part": 1,
  "parts": 1,
  "chars": 151,
  "text": "### Tomatoes\n\nPlant tomatoes after the last frost. Give them sun.\n\n```python\n..."
}

• heading_path: all the headings above this ch

... View full README on GitHub