MarkIndex

Vectorless, structure-first retrieval over large libraries of long Markdown documents — for agents.

MarkIndex parses every Markdown file into a tree of its own sections, stores the trees, the original lines and a mixed Chinese/English BM25 index in one SQLite file, and gives an agent five read-only tools to walk "library → folder → document → section" and read the original text with line numbers. No embeddings, no vector database, no chunking guesswork: the document's own structure is the index, and every piece of evidence can be cited back as [doc_path, L<start>-<end>].

Inspired by VectifyAI/PageIndex, MarkIndex is an independent, unofficial TypeScript implementation focused on Markdown. It is not affiliated with or endorsed by VectifyAI.

中文说明：README.zh-CN.md

Why

• Structure-first, not chunk-first. Long professional documents (filings, guidelines, laws, manuals) are already organized by their authors. Navigating that outline keeps context intact and makes answers explainable and citable; arbitrary chunks + vector similarity cut sections apart.
• Deterministic and LLM-free indexing. Parsing (mdast, not regex), tree building and the token-cost tree optimization never call a model; the same input always produces the same tree. LLMs are optional and only used for node summaries, document descriptions and the built-in retrieve() loop.
• Markdown done properly. Setext headings, preface text before the first heading, documents without any heading, front matter, # inside code fences, skipped heading levels, BOM / CRLF — every source line belongs to exactly one node, and oversized sections are split at block boundaries into chunk nodes.
• Keyword fallback that works for Chinese. BM25 over Intl.Segmenter word segmentation (Chinese, English and mixed text, identifiers like C++, ISO-27001, v2.1), plus a trigram substring index as a second pass.
• Budgeted tool responses. Tools never throw; they return JSON envelopes with next_steps guidance and stay under a 95 000-character budget, paginating large outlines and sections.

Install

npm install markindex          # library + CLI (Node.js >= 22.12)
npx markindex --help

From source:

git clone https://github.com/jackjin1997/markindex.git && cd markindex
npm install && npm run build
node dist/cli.js --help          # or `npm link` to get a `markindex` command

Optional, only for LLM features (summaries, retrieve(), agent-mode evaluation): install the AI SDK provider you want, e.g. npm install @ai-sdk/anthropic, and set its API key (ANTHROPIC_API_KEY, …). With npx and no local install, put the provider in the same temporary environment:

npx -p markindex -p @ai-sdk/anthropic markindex index ./docs --summary-model anthropic:claude-sonnet-5-5

CLI

# Build or incrementally update the index (stored in <root>/.markindex/index.db)
markindex index ./docs
markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 --estimate   # cost estimate, no LLM calls
markindex index ./docs --summary-model anthropic:claude-sonnet-5-5             # with node summaries
markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 --summary-rpm 50 --summary-tpm 40000   # rate-limited
markindex index ./docs --json                                                   # machine-readable report
markindex index ./docs --no-setext                           

... [View full README on GitHub](https://github.com/jackjin1997/markindex#readme)