Open source web context for AI agents
Reader is built for developers who want to understand, inspect, and build with the web context layer behind their AI agents.
Reader turns URLs into clean Markdown, structured JSON, crawl results, and browser sessions so agents can read, research, extract, and act on public web content. The project is built around a simple idea: AI agents need reliable web context, and developers should be able to see how that context is produced.
Why open source matters for web context
Web data is messy. A page may include useful content, navigation, scripts, ads, cookie banners, tracking code, repeated layout sections, hidden text, JavaScript rendered content, or inconsistent HTML. Before an AI agent can use that page, your system has to turn it into something readable, structured, and reliable.
For many teams, that web context layer becomes a black box. They send a URL to a tool, get text back, and hope the output is clean enough for an agent, retrieval system, extractor, or dataset pipeline.
Open source changes that relationship. With Reader, developers can inspect the ideas behind the system, understand the product model, test real URLs, review the output, and build workflows with more confidence. That matters because web context is not just an implementation detail. It is the source material your agent reasons over.
Reader's open source philosophy
Reader is not trying to be a black box scraping product. It is designed around a transparent developer workflow: URL or website to clean web context to Markdown, JSON, crawl results, or browser session to your agent, RAG pipeline, dataset, or application.
Clean outputs matter
Markdown quality, source URLs, and metadata are core to the product model. Structured extraction should be inspectable.
Start with the lightest tool
Crawls should be scoped and understandable. Browser sessions should be used when interaction is actually needed, not as a default.
Your application stays in control
Reader gives your system the web context. Your application decides what to do with it, including business logic, storage, alerts, and decisions.
The core capabilities
| Capability | What it does | Why it matters |
|---|---|---|
| Scrape | Turns one URL into clean page context | Give an agent readable source material |
| Crawl | Collects related pages from a website | Build site context for RAG, research, or datasets |
| Extract | Turns page content into structured JSON | Store fields your application can use |
| Browser | Opens a cloud browser session | Handle pages that need interaction |
Start with Scrape when you have one page. Use Crawl when you need many related pages. Use Extract when you need fields instead of full page text. Use Browser when the page requires interaction. Most workflows should start with the lightest capability that solves the problem.
Built for developers building agents
Reader is built for people who are building real AI workflows, not just testing scraping examples. Common use cases include:
AI agents that need to read live pages
RAG pipelines that need clean website content
Research agents that need source context
Product data workflows
Price monitoring systems
Competitive intelligence tools
Lead research workflows
Real estate listing extraction
LLM dataset collection
Documentation ingestion
Internal knowledge systems
These workflows all need the same foundation: useful web context from public pages. Instead of forcing developers to stitch together separate scraping, crawling, extraction, and browser tools, Reader provides the core capabilities in one web context API.
What developers can build with Reader
Agent web research
Give an agent a URL or website and let it collect clean source context before answering, summarizing, extracting, or planning.
RAG ingestion
Crawl public documentation, help centers, or knowledge bases into Markdown that can be chunked, embedded, and searched.
Product data extraction
Turn public product pages into structured records with source context attached.
Competitive intelligence
Store snapshots of public competitor pages and compare positioning, pricing, product, docs, and hiring changes over time.
Lead research
Collect public company website context and extract structured business details for account research.
Dataset preparation
Collect public pages as clean Markdown and metadata before filtering, reviewing, labeling, and versioning them in your own pipeline.
Browser fallback workflows
Use browser sessions when scraping does not capture the state your workflow needs.
Open source and hosted API together
Open source helps developers inspect, understand, and trust the system. A hosted API helps developers move faster without operating scraping, crawling, rendering, extraction, and browser infrastructure themselves.
Reader is designed to support a practical middle ground. Developers can understand the product model and build with the open source ecosystem, while still using a hosted API for production workflows that need reliability, credits, browser sessions, extraction, and developer tooling.
Open source
Inspect the system, understand the output model, test real URLs, review the output, and build with confidence. Your team should be able to test outputs, inspect the source context, and decide how Reader fits into your own architecture.
Hosted API
Move faster without operating infrastructure. Sign up at reader.dev. 1,000 free pages per month. Same engine, same output. Zero ops.
What open source does not replace
Open source is valuable, but it does not remove the need for product decisions. Reader gives your application clean web context, but your product still owns:
That separation is healthy. Reader gives you the web context layer. Your application owns the workflow.
When Reader is a good fit
Reader is especially useful when your AI system needs source context from the live web, but your team does not want to build and maintain the full web data layer from scratch.
Frequently asked questions
Build with an inspectable web context layer
Reader gives AI agents and data workflows clean web context from public pages, websites, structured fields, and browser sessions. Use it when your system needs to understand the web, not just retrieve HTML.