# scrapedatshi — RAG Pipeline API

> scrapedatshi is a pay-as-you-go API for building RAG (Retrieval-Augmented Generation) pipelines.
> Scrape URLs, chunk documents, generate embeddings, and inject into vector databases — all from a single API.

## What scrapedatshi does

scrapedatshi provides a complete RAG data pipeline:

1. **Scrape** — Convert any URL to clean Markdown (strips ads, nav bars, boilerplate)
2. **Chunk** — Split content into RAG-optimized segments (tables and code blocks never split mid-structure)
3. **Embed** — Generate vector embeddings via your own LLM/embedding provider key (BYOK)
4. **Inject** — Upsert vectors directly into your vector database
5. **Extract** — Pull structured data from any URL using your own LLM and a JSON schema

## Key Features

- **AutoRAG** — Crawl an entire site, chunk every page, embed, and inject in one API call. Large sites (>200 pages) are automatically batched server-side.
- **Contextual Retrieval (RAG 2.0)** — Per-chunk LLM context generation, proven to boost retrieval accuracy by 35–50%
- **Schema Extraction** — Extract structured JSON from any URL using your own LLM key
- **Multi-page Schema Extraction** — Crawl an entire domain and extract structured data from every page
- **JS Rendering** — Headless Chromium support for SPAs and JavaScript-heavy pages
- **Authenticated Scraping (v0.10.0+)** — Pass session cookies and custom headers for login-walled pages. Credentials are used locally on the user's machine and never transmitted to our servers.
- **Bulk Folder Ingest (v0.11.0+)** — Ingest an entire folder of pre-scraped files (.md, .txt, .json, .yaml, .yml) into a vector DB in one call. Automatically handles Scrapy JSON array exports — each item is extracted and ingested individually.
- **Project Scaffolding CLI (v0.11.0+)** — `scrapedatshi init my-project` generates a ready-to-run sandbox with pre-configured example scripts for every pipeline method, a .env template for all keys, and a provider discovery script that requires no API keys.
- **Expanded File Type Support (v0.12.0+)** — chunk_file, ingest, and ingest_folder now support CSV, XLSX, DOCX, IPYNB, HTML, XML, and all common code files (.py, .js, .ts, .sql, .go, .rb, .java, .cs, .cpp, .rs, etc.) in addition to PDF, MD, TXT, YAML, and JSON.
- **BYOK** — Bring your own LLM, embedding, and vector DB keys. scrapedatshi only charges for scraping and orchestration.
- **Claude MCP** — Use all tools directly inside Claude Desktop via the Model Context Protocol
- **Local Fetch Mode** — SDK/MCP default: URLs are fetched on the user's machine (their IP), not our server. Cheaper and more private.

## API Endpoints

- `GET /scrape` — Scrape any URL to clean Markdown
- `POST /v1/rag-chunk` — Scrape + chunk a URL into RAG-ready segments (supports `html=` for local-fetch mode)
- `POST /v1/crawl` — Crawl a domain's sitemap and return all page content
- `POST /v1/crawl-chunk` — Crawl + chunk an entire site, return JSON (no VDB)
- `POST /v1/spider` — Deep spider crawl: follow links from a root URL (BFS, server-side)
- `POST /v1/autorag` — Full pipeline: crawl site → chunk → embed → inject into vector DB
- `POST /v1/sync` — Full pipeline for a single URL: scrape → chunk → embed → inject (supports `html=` for local-fetch mode)
- `POST /v1/ingest` — Full pipeline for local files: parse → chunk → embed → inject
- `POST /v1/ingest-chunk` — Parse + chunk local files, return JSON (no VDB)
- `POST /v1/extract` — Extract structured data from a URL using your LLM
- `POST /v1/extract-crawl` — Multi-page schema extraction via site crawl
- `POST /v1/process-html` — Process pre-fetched HTML (used by SDK local-fetch mode)
- `POST /v1/process-text` — Process pre-extracted text (used by SDK local-fetch mode for files)
- `POST /portal/pdf/extract` — Extract text or tables from a PDF (authenticated, billed; URL or file upload)
- `POST /v1/inspect-vectordb` — Read vector DB metadata: dimension, vector count, suggested embedding models (free)
- `POST /v1/query` — Semantic search: embed a query and retrieve the most relevant chunks from your vector DB
- `POST /v1/rag-chat` — RAG Chat: embed a query, retrieve top-N chunks from your vector DB, and generate a grounded LLM answer

## Supported Providers

**Embedding:** OpenAI, Cohere, Google Gemini, Mistral, Voyage AI, Ollama (local)

**Vector Databases:** Pinecone, Qdrant, Supabase (pgvector), Weaviate, MongoDB Atlas, Azure Cosmos DB, ChromaDB, LanceDB

**LLM (for extraction + contextual retrieval):** OpenAI, Anthropic, Google Gemini

## Pricing

Pay-as-you-go credit wallet. No subscription required.

- URL Fetch (local, SDK/MCP default): $0.0020 / URL
- URL Fetch (server): $0.0040 / URL
- Spider Fetch (server): $0.0050 / URL (deep crawl)
- Chunk Fee: $0.0005 / chunk
- Injection Fee: $0.0030 / chunk (vector DB upserts)
- Contextual Retrieval: $0.0010 / enriched chunk
- JS Render: $0.0050 / URL
- Schema Extract: $0.0030 + ($0.0001 × fields) per page
- PDF Extract (file upload): $0.0020 / file — processing fee, no proxy cost
- PDF Extract (URL fetch): $0.0040 / URL — server fetches the PDF
- Vector Query: $0.0002 / chunk retrieved (/v1/query, /v1/rag-chat)
- Inspect Vector DB: free (/v1/inspect-vectordb)

New accounts receive $1.00 free credits. No credit card required.

## Python SDK

Install: `pip install scrapedatshi`

PyPI: https://pypi.org/project/scrapedatshi/
GitHub: https://github.com/scrapedatshi/scrapedatshi-py

```python
from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()  # reads SCRAPEDATSHI_API_KEY from env

# Chunk a URL
result = client.pipeline.chunk_url("https://docs.example.com")
print(f"Got {result.total_chunks} chunks")

# Authenticated scraping (v0.10.0+) — cookies stay on your machine
result = client.pipeline.chunk_url(
    "https://internal.company.com/wiki",
    cookies={"session": "abc123"},
    headers={"Authorization": "Bearer eyJ..."},
)

# Authenticated crawl with subdomain scope
result = client.pipeline.crawl(
    "https://company.com",
    cookies={"session": "abc123"},
    allow_subdomains=True,
    max_pages=30,
)

# Full AutoRAG pipeline
result = client.pipeline.autorag(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
print(f"Injected {result.vectors_upserted} vectors")

# Schema extraction
result = client.pipeline.extract(
    url="https://example.com/products/widget",
    schema={"title": "string — product name", "price": "number — price in USD"},
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
)
print(result.extracted)

# RAG Chat
result = client.pipeline.rag_chat(
    query="How do I authenticate with the API?",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
)
print(result.answer)
```

## Claude MCP Server

Use all scrapedatshi tools directly inside Claude Desktop — no code required.

Install: `pip install scrapedatshi-mcp`

PyPI: https://pypi.org/project/scrapedatshi-mcp/
GitHub: https://github.com/scrapedatshi/scrapedatshi-mcp

Claude Desktop config (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "scrapedatshi": {
      "command": "uvx",
      "args": [
        "--from", "scrapedatshi-mcp[all]",
        "--refresh",
        "scrapedatshi-mcp"
      ],
      "env": {
        "SCRAPEDATSHI_API_KEY": "your-api-key"
      }
    }
  }
}
```

Available MCP tools: `scrape_url`, `chunk_file`, `crawl_site`, `extract_data`, `extract_crawl`, `sync_to_vectordb`, `ingest_file`, `ingest_folder`, `autorag`, `inspect_vectordb`, `query_vectordb`, `rag_chat`, `verify_provider_key`, `list_embedding_providers`, `list_vector_db_providers`, `get_usage_guide`

## Links

- Website: https://scrapedatshi.com
- API Docs: https://scrapedatshi.com/dev
- Sign up: https://scrapedatshi.com/portal/register
- Billing: https://scrapedatshi.com/portal/billing
- Python SDK (PyPI): https://pypi.org/project/scrapedatshi/
- Python SDK (GitHub): https://github.com/scrapedatshi/scrapedatshi-py
- Claude MCP (PyPI): https://pypi.org/project/scrapedatshi-mcp/
- Claude MCP (GitHub): https://github.com/scrapedatshi/scrapedatshi-mcp