HTML API Reader
A high-performance REST API server written in Rust that fetches and extracts content from web pages. Built using Clean Architecture principles with a workspace structure.
Table of Contents
Quick Start
🚀 Using Local Development
-
Build and run:
cargo build --release
cargo run --bin html-mcp-reader
-
Test the API:
# Health check
curl http://localhost:8085/health
# Fetch web content
curl -X POST http://localhost:8085/api/fetch \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
🐳 Using Docker
-
Build the Docker image:
docker build -t html-api-reader:latest .
-
Run the container:
docker run -p 8085:8085 html-api-reader:latest
-
Or use Docker Compose:
docker-compose up
Features
- REST API: Simple HTTP endpoints for web content fetching
- HTML Content Extraction: Extract text content from HTML pages
- Flexible Options: Configure text extraction, redirects, timeouts, and user agents
- Clean Architecture: Separated concerns with domain-driven design
- Async/Await: High-performance async processing with Tokio
- CORS Support: Cross-origin requests enabled
- Health Monitoring: Built-in health check endpoint
- Docker Ready: Containerized deployment with health checks
API Endpoints
GET /health
Returns the health status of the API server.
Response:
{
"status": "healthy",
"version": "0.1.0"
}
POST /api/fetch
Fetches and extracts content from web pages.
Request Body:
{
"url": "https://example.com",
"extract_text_only": true,
"follow_redirects": true,
"timeout_seconds": 30,
"user_agent": "html-api-reader/0.1.0"
}
Parameters:
url (required): The URL to fetch content fromextract_text_only (optional, default: true): Whether to extract only text contentfollow_redirects (optional, default: true): Whether to follow HTTP redirectstimeout_seconds (optional, default: 30, max: 300): Request timeout in secondsuser_agent (optional): Custom User-Agent header
Response:
{
"url": "https://example.com",
"title": "Example Domain",
"text_content": "Example Domain This domain is for use in illustrative examples...",
"raw_html": "<!doctype html><html>...",
"metadata": {
"content_type": "text/html; charset=utf-8",
"status_code": 200,
"content_length": 1256,
"last_modified": null,
"charset": null
}
}
Error Response:
{
"error": "INVALID_URL",
"message": "URL cannot be empty"
}
Architecture
The project follows Clean Architecture principles with these layers:
- Domain: Core business logic and interfaces (
domain/)
- Application: Use cases and business services (
application/)
- Infrastructure: External adapters for HTTP, HTML parsing, and REST API (
infrastructure/)
- Runner: Entry point and dependency injection (
runner/)
Dependencies
Key dependencies used:
axum: Modern web framework for the REST APItower-http: HTTP middleware (CORS support)reqwest: HTTP client for fetching web contentscraper: HTML parsing and text extractionserde/serde_json: JSON serialization for API requests/responsestracing: Structured loggingtokio: Async runtime
Building
Local Development
# Build debug version
cargo build
# Build release version (optimized)
cargo build --release
# Run tests
cargo test
# Check code quality
cargo clippy
cargo fmt
Docker
# Build the Docker image
... [View full README on GitHub](https://github.com/MauricioZapata00/250812-html-reader-mcp-server#readme)
