CLI Usage

Command-line access to all Xberg extraction features.

Installation

Install Script (Linux/macOS)

Bash

curl -fsSL https://raw.githubusercontent.com/xberg-io/xberg/main/scripts/install.sh | bash

Homebrew (macOS/Linux)

Bash

brew trust xberg-io/tap
brew install xberg-io/tap/xberg

Cargo (Cross-platform)

Bash

cargo install xberg-cli

Docker

Bash

docker pull ghcr.io/xberg-io/xberg-cli:latest
docker run -v $(pwd):/data ghcr.io/xberg-io/xberg-cli:latest extract /data/document.pdf

Go (SDK)

Bash

go get github.com/xberg-io/xberg/packages/go@latest

Feature Availability

Homebrew Installation:

• ✅ Text extraction (PDF, Office, images, 106 formats)
• ✅ OCR with Tesseract
• ✅ HTTP API server (serve command)
• ✅ MCP protocol server (mcp command)
• ✅ Chunking, quality scoring, language detection
• ❌ Embeddings - Not available via CLI flags. Use config file or Docker image.

Docker Images:

• All features enabled including embeddings (ONNX Runtime included)

Global Flags

Log Level

--log-level controls log verbosity and overrides RUST_LOG.

Terminal

# Set log level to debug for troubleshooting
xberg --log-level debug extract document.pdf

# Suppress all but error messages
xberg --log-level error batch documents/*.pdf

# Trace-level logging for maximum detail
xberg --log-level trace extract document.pdf

Valid levels: trace, debug, info (default), warn, error.

Colored Output

Output is colored by default. Disable with NO_COLOR:

Terminal

# Disable colored output
NO_COLOR=1 xberg extract document.pdf

Basic Usage

Extract from Single File

Terminal

# Extract text content to stdout
xberg extract document.pdf

# Specify MIME type (auto-detected if not provided)
xberg extract document.pdf --mime-type application/pdf

Batch Extract Multiple Files

Terminal

# Extract from multiple files
xberg batch doc1.pdf doc2.docx doc3.txt

# Batch extract all PDFs in directory
xberg batch documents/*.pdf

# Batch extract recursively
xberg batch documents/**/*.pdf

Output Formats

Terminal

# Output as plain text (default for extract)
xberg extract document.pdf --format text

# Output as JSON (default for batch)
xberg batch documents/*.pdf --format json

# Extract single file as JSON
xberg extract document.pdf --format json

# Output as TOON wire format (token-efficient alternative to JSON)
xberg extract document.pdf --format toon

Content Output Format

--content-format (alias: --output-format) sets the format of extracted text content:

Terminal

# Extract as plain text (default)
xberg extract document.pdf --content-format plain

# Extract as Markdown
xberg extract document.pdf --content-format markdown

# Extract as Djot markup
xberg extract document.pdf --content-format djot

# Extract as HTML
xberg extract document.pdf --content-format html

# Combine content format with wire format
xberg extract document.pdf --content-format markdown --format toon

--content-format formats result.content; --format controls the wire format of the entire response (text, json, or toon).

OCR Extraction

Enable OCR

Terminal

# Enable OCR (overrides config file setting)
xberg extract scanned.pdf --ocr true

# Disable OCR
xberg extract document.pdf --ocr false

Force OCR

Force OCR even for PDFs with text layer:

Terminal

# Force OCR to run regardless of existing text
xberg extract document.pdf --force-ocr true

OCR Language Selection

--ocr-language is backend-agnostic and overrides config-file or default settings.

Backend	Code format	Examples
Tesseract	ISO 639-3 (three-letter)	eng, fra, deu, spa, jpn
PaddleOCR	short codes / language names	en, ch, french, korean, thai, cyrillic

Terminal

# French OCR with Tesseract (default backend)
xberg extract --ocr true --ocr-language fra document.pdf

# Chinese OCR with PaddleOCR
xberg extract --ocr true --ocr-backend paddle-ocr --ocr-language ch document.pdf

# Thai OCR with PaddleOCR
xberg extract --ocr true --ocr-backend paddle-ocr --ocr-language thai document.pdf

# German OCR with Tesseract
xberg extract --ocr true --ocr-language deu document.pdf

# Override config file language with Spanish
xberg extract document.pdf --config xberg.toml --ocr-language spa

OCR Configuration

OCR options live in the config file; CLI flags override:

Terminal

xberg extract scanned.pdf --config xberg.toml --ocr true

See Configuration Files for backend, language, and Tesseract options.

Configuration Files

Using Config Files

Xberg auto-discovers xberg.toml by walking up from the current directory. For YAML or JSON, pass --config explicitly.

Terminal

xberg extract document.pdf  # auto-discovers xberg.toml

Specify Config File

Load TOML, YAML (.yaml/.yml), or JSON via --config:

Terminal

xberg extract document.pdf --config my-config.toml
xberg extract document.pdf --config xberg.yaml
xberg extract document.pdf --config my-config.json

Inline JSON Config

Inline JSON is merged after config file, before individual flags:

Terminal

# Inline JSON (applied after config file)
xberg extract document.pdf --config-json '{"ocr":{"backend":"tesseract"},"chunking":{"max_chars":1000}}'

# Base64-encoded JSON (useful in shells where quoting is awkward)
xberg extract document.pdf --config-json-base64 eyJvY3IiOnsiYmFja2VuZCI6InRlc3NlcmFjdCJ9fQ==

Both extract and batch support --config-json and --config-json-base64.

Example Config Files

xberg.toml:

OCR configuration

use_cache = true
enable_quality_processing = true

[ocr]
backend = "tesseract"
language = "eng"

[ocr.tesseract_config]
psm = 3

[chunking]
max_characters = 1000
overlap = 100

xberg.yaml:

xberg.yaml

use_cache: true
enable_quality_processing: true

ocr:
  backend: tesseract
  language: eng
  tesseract_config:
    psm: 3

chunking:
  max_characters: 1000
  overlap: 100

xberg.json:

xberg.json

{
  "use_cache": true,
  "enable_quality_processing": true,
  "ocr": {
    "backend": "tesseract",
    "language": "eng",
    "tesseract_config": {
      "psm": 3
    }
  },
  "chunking": {
    "max_characters": 1000,
    "overlap": 100
  }
}

Batch Processing

Process multiple files with batch:

Terminal

# Extract all PDFs in directory
xberg batch documents/*.pdf

# Extract PDFs recursively from subdirectories
xberg batch documents/**/*.pdf

# Extract multiple file types
xberg batch documents/**/*.{pdf,docx,txt}

Batch with Output Formats

Terminal

# Output as JSON (default for batch command)
xberg batch documents/*.pdf --format json

# Output as plain text
xberg batch documents/*.pdf --format text

Batch with OCR

Terminal

# Batch extract with OCR enabled
xberg batch scanned/*.pdf --ocr true

# Batch extract with force OCR
xberg batch documents/*.pdf --force-ocr true

# Batch extract with quality processing
xberg batch documents/*.pdf --quality true

Batch with Content Format

Terminal

# Batch extract with djot formatting
xberg batch documents/*.pdf --output-format djot --format json

# Batch extract as Markdown
xberg batch documents/*.pdf --output-format markdown --format json

# Batch extract as HTML
xberg batch documents/*.pdf --output-format html --format json

Advanced Features

Language Detection

Terminal

# Extract with automatic language detection
xberg extract document.pdf --detect-language true

# Disable language detection
xberg extract document.pdf --detect-language false

Content Chunking

Terminal

# Split content into chunks for LLM processing
xberg extract document.pdf --chunk true

# Specify chunk size and overlap
xberg extract document.pdf --chunk true --chunk-size 1000 --chunk-overlap 100

# Output chunked content as JSON
xberg extract document.pdf --chunk true --format json

Quality Processing

Terminal

# Apply quality processing for improved formatting
xberg extract document.pdf --quality true

# Disable quality processing
xberg extract document.pdf --quality false

# Batch extraction with quality processing
xberg batch documents/*.pdf --quality true

Caching

Terminal

# Extract with result caching enabled (default)
xberg extract document.pdf

# Extract without caching results
xberg extract document.pdf --no-cache true

# Clear all cached results
xberg cache clear

# View cache statistics
xberg cache stats

Tree-sitter Grammars

Manage the grammar cache used for code intelligence. This command is available when the CLI is built with the tree-sitter feature.

Terminal

# Download selected grammars
xberg tree-sitter download python rust go

# Download configured language groups
xberg tree-sitter download --from-config

# Inspect or clear the grammar cache
xberg tree-sitter list --downloaded
xberg tree-sitter cache-dir
xberg tree-sitter clean

Environment Diagnostics

doctor checks whether the backends in your config will actually run on this machine, before the first document. Each check reports pass, warn, fail, or skip with a one-line reason; warnings are actionable but never fail the command, and it exits nonzero only on failures.

Terminal

# Probe the backends from xberg.toml (or the discovered config)
xberg doctor

# JSON output for bug reports
xberg doctor --format json

# Also remove stray files from xberg-owned cache dirs
xberg doctor --clean

Tesseract checks tessdata per configured language, PaddleOCR verifies model checksums, VLM checks the API key and endpoint reachability (no billable call), and layout detection runs one real RT-DETR inference. Models that aren’t downloaded yet report skip rather than failing.

When XBERG_CACHE_DIR is set, cache inspection and --clean are disabled (reported as skip): the override is a raw path and xberg cannot verify it owns the directory.

Extraction Override Flags

extract and batch accept the flags below; they take precedence over config-file settings.

OCR Flags

Flag	Description
--ocr <true|false>	Enable or disable OCR. Defaults to tesseract backend when enabled.
--ocr-backend <BACKEND>	OCR backend: tesseract, paddle-ocr, sceptre, candle-trocr, candle-paddleocr-vl, candle-paddleocr-vl-15, candle-glm-ocr, candle-deepseek-ocr, or vlm.
--ocr-language <LANG>	OCR language code. Sceptre accepts its eight group tokens or ISO aliases such as eng, deu, tel, and kan.
--force-ocr <true|false>	Force OCR even if the document has an existing text layer.
--ocr-auto-rotate <true|false>	Automatically rotate images before OCR based on detected orientation.
--disable-ocr <true|false>	Disable OCR entirely, even for images.

Candle-based backends (candle-trocr, candle-paddleocr-vl, candle-paddleocr-vl-15, candle-glm-ocr, candle-deepseek-ocr) are pure-Rust VLM and vision-transformer OCR engines. No ONNX Runtime required; GPU-accelerated on Metal (macOS) and CUDA (Linux). They ship compiled into the CLI/Docker image by default — no extra install or feature flag needed. Model weights download automatically from Hugging Face on first use.

Terminal

xberg extract scanned.pdf --ocr true --ocr-backend paddle-ocr --ocr-language ch
xberg extract document.pdf --force-ocr true --ocr-auto-rotate true

Chunking Flags

Flag	Description
--chunk <true|false>	Enable or disable text chunking.
--chunk-size <N>	Maximum chunk size in characters (default: 1000).
--chunk-overlap <N>	Overlap between consecutive chunks in characters (default: 200).
--chunking-tokenizer <MODEL>	Tokenizer model for token-based chunk sizing (for example Xenova/gpt-4o). Implicitly enables chunking. Requires the chunking-tokenizers feature.

Terminal

xberg extract document.pdf --chunk true --chunk-size 512 --chunk-overlap 50
xberg extract document.pdf --chunking-tokenizer "Xenova/gpt-4o"

Output Flags

Flag	Description
--content-format <FORMAT>	Content output format: plain, markdown, djot, html, json, or doctags. Controls how extracted text is formatted. (Deprecated alias: --output-format)
--include-structure <true|false>	Include hierarchical document structure in results.

Terminal

xberg extract document.pdf --content-format markdown --include-structure true

Layout Detection Flags

Flag	Description
--layout	Enable layout detection with default settings (RT-DETR v2). Use --layout false to explicitly disable. Requires the layout-detection feature.
--layout-confidence <FLOAT>	Layout detection confidence threshold (0.0 - 1.0).
--layout-table-model <MODEL>	Table structure model: tatr (default), slanet_wired, slanet_wireless, slanet_plus, slanet_auto, disabled.

Terminal

xberg extract document.pdf --layout --layout-confidence 0.7

Acceleration Flags

Flag	Description
--acceleration <PROVIDER>	ONNX Runtime execution provider for model inference: auto, cpu, coreml, cuda, or tensorrt.

Terminal

# Use CoreML on macOS for GPU acceleration
xberg extract document.pdf --acceleration coreml

# Use CUDA on Linux with NVIDIA GPU
xberg extract document.pdf --acceleration cuda

Page Flags

Flag	Description
--extract-pages <true|false>	Extract pages as a separate array in results.
--page-markers <true|false>	Insert page marker comments into the main content string.

Terminal

xberg extract document.pdf --extract-pages true --page-markers true --format json

Image Flags

Flag	Description
--extract-images <true|false>	Enable image extraction from documents.
--target-dpi <N>	Target DPI for image normalisation (36 - 2400).

Terminal

xberg extract document.pdf --extract-images true --target-dpi 300

PDF Flags

Flag	Description
--pdf-password <PASSWORD>	Password for encrypted PDFs. Can be specified multiple times for multiple passwords.
--pdf-extract-images <true|false>	Extract images embedded in PDF pages.
--pdf-extract-metadata <true|false>	Extract PDF metadata (title, author, etc.).

Terminal

xberg extract encrypted.pdf --pdf-password "secret"
xberg extract document.pdf --pdf-extract-images true --pdf-extract-metadata true

Token Reduction Flags

Flag	Description
--token-reduction <LEVEL>	Token reduction intensity: off, light, moderate, aggressive, or maximum. Reduces token count for LLM consumption.

Terminal

# Aggressive token reduction for cheaper LLM processing
xberg extract document.pdf --token-reduction aggressive

# Maximum compression (lossy)
xberg extract document.pdf --token-reduction maximum

Quality and Detection Flags

Flag	Description
--quality <true|false>	Enable quality post-processing for improved formatting.
--detect-language <true|false>	Enable automatic language detection on extracted text.

Cache Flags

Flag	Description
--no-cache <true|false>	Disable extraction result caching.
--cache-namespace <NAMESPACE>	Cache namespace for tenant isolation.
--cache-ttl-secs <SECONDS>	Per-request cache TTL in seconds (0 = skip cache).

Concurrency Flags

Flag	Description
--max-concurrent <N>	Limit parallel extractions in batch mode.
--max-threads <N>	Cap all internal thread pools (Rayon, ONNX intra-op, batch semaphore). Useful for constrained environments.

Terminal

xberg batch documents/*.pdf --max-concurrent 4 --max-threads 8

Email Flags

Flag	Description
--msg-codepage <N>	Windows codepage fallback for MSG files without codepage metadata. Common values: 1250 (Central European), 1251 (Cyrillic), 1252 (Western).

Terminal

xberg extract message.msg --msg-codepage 1251

Output Options

Standard Output (Text Format)

Terminal

# Extract and print content to stdout
xberg extract document.pdf

# Extract and redirect output to file
xberg extract document.pdf > output.txt

# Batch extract as text
xberg batch documents/*.pdf --format text

JSON Output

Terminal

# Output as JSON
xberg extract document.pdf --format json

# Batch extract as JSON (default format)
xberg batch documents/*.pdf --format json

JSON Output Structure:

JSON Response

{
  "content": "Extracted text content...",
  "metadata": {
    "mime_type": "application/pdf"
  }
}

Error Handling

The CLI returns non-zero exit codes on error. Use shell idioms:

Terminal

# Check for extraction errors
xberg extract document.pdf || echo "Extraction failed"

# Continue processing even if one file fails (bash)
for file in documents/*.pdf; do
  xberg batch "$file" || continue
done

Examples

Extract Single PDF

Extract text from PDF

xberg extract document.pdf

Batch Extract All PDFs in Directory

Extract all PDFs from directory as JSON

xberg batch documents/*.pdf --format json

OCR Scanned Documents

OCR extraction from scanned documents

xberg batch scans/*.pdf --ocr true --format json

Extract with Quality Processing

Extract with quality processing enabled

xberg extract document.pdf --quality true --format json

Extract with Chunking

Extract with chunking for LLM processing

xberg extract document.pdf --config xberg.toml --chunk true --chunk-size 1000 --chunk-overlap 100 --format json

Batch Extract Multiple File Types

Extract multiple file types in batch

xberg batch documents/**/*.{pdf,docx,txt} --format json

Extract with Config File

Extract using configuration file

xberg extract document.pdf --config /path/to/xberg.toml

Detect MIME Type

Detect file MIME type

xberg detect document.pdf

Docker Usage

Use ghcr.io/xberg-io/xberg-cli:latest for the CLI image, or ghcr.io/xberg-io/xberg:latest for the full image (also includes the CLI).

Basic Docker

Terminal

# Extract document using Docker with mounted directory
docker run -v $(pwd):/data ghcr.io/xberg-io/xberg-cli:latest \
  extract /data/document.pdf

# Extract and save output to host directory using shell redirection
docker run -v $(pwd):/data ghcr.io/xberg-io/xberg-cli:latest \
  extract /data/document.pdf > output.txt

Docker with OCR

Terminal

# Extract with OCR using Docker
docker run -v $(pwd):/data ghcr.io/xberg-io/xberg-cli:latest \
  extract /data/scanned.pdf --ocr true

Docker Compose

docker-compose.yaml:

docker-compose.yaml

version: "3.8"

services:
  xberg:
    image: ghcr.io/xberg-io/xberg-cli:latest
    volumes:
      - ./documents:/input
    command: extract /input/document.pdf --ocr true

Run:

Terminal

docker-compose up

Performance Tips

Optimize Extraction Speed

Terminal

# Extract without quality processing for faster speed
xberg extract large.pdf --quality false

# Use batch for processing multiple files
xberg batch large_files/*.pdf --format json

Manage Memory Usage

Terminal

# Disable caching to reduce memory footprint
xberg extract large_file.pdf --no-cache true

# Compress output to save disk space
xberg extract document.pdf | gzip > output.txt.gz

Troubleshooting

Check Installation

Terminal

# Display installed version
xberg --version

# Display help for commands
xberg --help

Common Issues

Issue: “Tesseract not found”

When using OCR, Tesseract must be installed:

Terminal

# Install Tesseract OCR engine on macOS
brew install tesseract

# Install Tesseract OCR engine on Ubuntu
sudo apt-get install tesseract-ocr

Issue: “File not found”

Ensure the file path is correct and accessible:

Terminal

# Check if file exists and is readable
ls -la document.pdf

# Extract with absolute path
xberg extract /absolute/path/to/document.pdf

Server Commands

Start API Server

serve starts the HTTP REST API:

Terminal

# Start server on default host (127.0.0.1) and port (8000)
xberg serve

# Start server on specific host and port (-H / -p are short forms)
xberg serve --host 0.0.0.0 --port 8000
xberg serve -H 0.0.0.0 -p 8000

# Start server with custom configuration file
xberg serve --config xberg.toml --host 0.0.0.0 --port 8000

Server Endpoints

The server provides the following endpoints:

• POST /extract - Extract text from uploaded files
• POST /batch - Batch extract from multiple files
• GET /detect - Detect MIME type of file
• GET /health - Health check
• GET /info - Server information
• GET /cache/stats - Cache statistics
• POST /cache/clear - Clear cache

See API Server Guide for full API details.

Start MCP Server

mcp starts a Model Context Protocol server for AI agents:

Terminal

# Start MCP server with stdio transport (default for Claude Desktop)
xberg mcp

# Start MCP server with HTTP transport
xberg mcp --transport http

# Start MCP server on specific HTTP host and port
xberg mcp --transport http --host 0.0.0.0 --port 8001

# Start MCP server with custom configuration file
xberg mcp --config xberg.toml --transport stdio

The MCP server provides tools for AI agents:

• extract - Extract text from a file path
• extract - Extract text from base64-encoded bytes
• extract_batch - Extract from multiple files

See API Server Guide for MCP integration details.

Embeddings

Generate vector embeddings using pre-trained models. Input via --text or stdin.

Terminal

# Generate embeddings for a single text
xberg embed --text "hello world" --preset balanced

# Generate embeddings with a specific preset
xberg embed --text "document content" --preset fast

# Batch embed multiple texts
xberg embed --text "first document" --text "second document" --preset quality

# Read from stdin
echo "hello world" | xberg embed --preset balanced

# Output as text instead of JSON
xberg embed --text "hello" --preset balanced --format text

Available presets: fast, balanced (default), quality, multilingual.

Feature Availability

The embed command requires the embeddings feature. It is available in Docker images but not in Homebrew installations.

Chunking Command

Split text with configurable size and overlap. Input via --text or stdin.

Terminal

# Chunk text with default settings
xberg chunk --text "long text content to be split into chunks..."

# Specify chunk size and overlap
xberg chunk --text "long text..." --chunk-size 512 --chunk-overlap 50

# Use markdown-aware chunking
xberg chunk --text "# Heading\n\nParagraph..." --chunker-type markdown

# Use a tokenizer model for token-based sizing
xberg chunk --text "long text..." --chunking-tokenizer "Xenova/gpt-4o"

# Read from stdin
cat document.txt | xberg chunk --chunk-size 1000

# Output as text instead of JSON
xberg chunk --text "long text..." --format text

# Use a config file for chunking settings
xberg chunk --text "long text..." --config xberg.toml

Shell Completions

Tab-completion scripts for bash, zsh, and fish:

Terminal

# Generate bash completions
xberg completions bash

# Generate zsh completions
xberg completions zsh

# Generate fish completions
xberg completions fish

# Install bash completions
eval "$(xberg completions bash)"

# Install zsh completions (add to .zshrc)
eval "$(xberg completions zsh)"

API Utilities

Dump OpenAPI Schema

Output the OpenAPI 3.1 specification — useful for code generation and API client tooling.

Terminal

# Print OpenAPI schema as JSON
xberg api schema

# Save to file
xberg api schema > openapi.json

Feature Availability

The api subcommand requires the api feature.

List Supported Formats

List supported formats with extensions and MIME types:

Terminal

# List formats as a table
xberg formats

# List formats as JSON
xberg formats --format json

Cache Management

View Cache Statistics

Terminal

# Display cache usage statistics
xberg cache stats

# Display statistics for specific cache directory
xberg cache stats --cache-dir /path/to/cache

# Output cache statistics as JSON
xberg cache stats --format json

Clear Cache

Terminal

# Remove all cached extraction results
xberg cache clear

# Clear specific cache directory
xberg cache clear --cache-dir /path/to/cache

# Clear cache and display removal details
xberg cache clear --format json

Warm Model Cache

Pre-download ML models (PaddleOCR, layout detection, embeddings, NER) for offline use — useful for containerized deployments.

Default cache directories:

• Linux: ~/.cache/xberg/{module} (or $XDG_CACHE_HOME/xberg/{module})
• macOS: ~/Library/Caches/xberg/{module}
• Windows: %LOCALAPPDATA%/xberg/{module}

Override with XBERG_CACHE_DIR or --cache-dir.

NER warming downloads exported GLiNER artifacts from xberg-io/gliner-models, not arbitrary GLiNER source repositories. If that Hugging Face repository is private or not publicly readable, configure credentials supported by hf-hub first.

Terminal

# Download all OCR and layout models eagerly
xberg cache warm

# Download to a specific cache directory
xberg cache warm --cache-dir /path/to/cache

# Also download all 4 embedding model presets (fast, balanced, quality, multilingual)
xberg cache warm --all-embeddings

# Download a specific embedding model preset
xberg cache warm --embedding-model balanced

# Download the default GLiNER NER model alias
xberg cache warm --ner

# Download a specific xberg GLiNER alias or catalog id
xberg cache warm --ner-model fast

# Output download results as JSON
xberg cache warm --format json

Model Manifest

Manifest of expected model files with SHA256 checksums and sizes — for cache integrity checks or scripted pre-population.

Terminal

# Output manifest as JSON (default)
xberg cache manifest

# Output manifest as human-readable text
xberg cache manifest --format text

Getting Help

CLI Help

Terminal

# Display general CLI help
xberg --help

# Display command-specific help
xberg extract --help
xberg batch --help
xberg detect --help
xberg formats --help
xberg version --help
xberg embed --help
xberg chunk --help
xberg completions --help
xberg serve --help
xberg mcp --help
xberg cache --help
xberg cache stats --help
xberg cache clear --help
xberg cache warm --help
xberg cache manifest --help
xberg api schema --help

Version Information

Terminal

# Display version number
xberg --version

# Show version with JSON output
xberg version --format json

Next Steps

• API Server Guide - API and MCP server setup
• Chunking - Split text for RAG
• Embeddings - Semantic vectors for search
• Language Detection - Multilingual document analysis
• Token Reduction - Optimize for LLMs
• Quality Processing - Filter low-quality text
• PDF Form Fields - Extract form data
• Plugin Development - Extend Xberg functionality
• API Reference - Programmatic access