rag_engine

rag_engine is a Rust-first retrieval engine for RAG pipelines. It provides SQLite-backed source/chunk storage, HNSW vector search, BM25 keyword search, hybrid fusion search, tokenization, and document parsing.

Default mode is pure Rust. Flutter Rust Bridge is optional via the frb-bridge feature.

Feature Matrix

• default (no extra features): pure Rust core APIs
• frb-bridge: enables FRB exports and generated bridge module
• vector_faer: faer-backed vector math backend
• vector_quant_i8: i8 quantization path for embeddings

Install

[dependencies]
rag_engine = "0.8.1"

With FRB:

[dependencies]
rag_engine = { version = "0.8.1", features = ["frb-bridge"] }

Quick Start (Rust Core)

The minimal startup order is:

1. Initialize core logger
2. Initialize DB pool
3. Initialize source/chunk schema
4. Add sources and chunks (with your own embeddings)
5. Search chunks

use rag_engine::api::{
    db_pool,
    semantic_chunker,
    simple,
    source_rag::{self, ChunkData},
};

fn embed(text: &str) -> Vec<f32> {
    // Plug in your embedding model here (OpenAI, local model, etc.).
    // Keep dimension consistent across all chunks and queries.
    let _ = text;
    vec![0.0; 384]
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    simple::init_core();

    db_pool::init_db_pool("./rag.sqlite3".to_string(), 4)?;
    source_rag::init_source_db()?;

    let text = "RAG combines retrieval and generation.\n\nHNSW accelerates vector search.";
    let source = source_rag::add_source(
        text.to_string(),
        Some("{\"origin\":\"quickstart\"}".to_string()),
        Some("intro".to_string()),
    )?;

    let semantic_chunks = semantic_chunker::semantic_chunk(text.to_string(), 600);
    let chunks: Vec<ChunkData> = semantic_chunks
        .into_iter()
        .map(|c| ChunkData {
            content: c.content.clone(),
            chunk_index: c.index,
            start_pos: c.start_pos,
            end_pos: c.end_pos,
            chunk_type: c.chunk_type,
            embedding: embed(&c.content),
        })
        .collect();

    source_rag::add_chunks(source.source_id, chunks)?;

    let query = "what is hnsw";
    let results = source_rag::search_chunks(embed(query), 5)?;
    for r in results {
        println!("[score={:.4}] {}", r.similarity, r.content);
    }

    Ok(())
}

Onboarding Notes

• rag_engine does not generate embeddings for you.
• You must pass vectors from your own embedding model into add_chunks(...).
• Query vectors must have the same dimension as indexed chunk vectors.
• DB pool must be initialized before using source/chunk APIs.

Common API Entry Points

• api::simple::init_core: core bootstrap
• api::db_pool::init_db_pool: initialize SQLite pool
• api::source_rag::init_source_db: create/migrate source/chunk schema
• api::source_rag::add_source: register source metadata/content
• api::source_rag::add_chunks: insert chunk payload + embeddings
• api::source_rag::search_chunks: vector retrieval
• api::hybrid_search::search_hybrid: vector + BM25 fused retrieval
• api::document_parser::*: PDF/DOCX text extraction helpers
• api::tokenizer::*: tokenizer init/tokenize/decode utilities

Flutter Bridge

With frb-bridge enabled, FRB-facing exports are compiled, including:

• FRB init path (api::simple::init_app)
• Dart log stream bridge functions
• FRB attributes for compatible public API surfaces

Core logic remains usable without FRB.

Production Checklist

• Use persistent DB path (not temp file) and back it up
• Keep embedding dimension fixed across indexing and querying
• Rebuild indexes if you bulk-update chunks/sources
• Set logging according to runtime requirements

Roadmap

• Keep current SQLite-first architecture as the default stable path
• Add storage abstraction layer (Storage / VectorStore traits) to decouple core logic from SQLite internals
• Add optional backends for server deployments:
  • pgvector backend
  • qdrant backend
• Define backend compatibility contract before rollout:
  • fixed embedding dimension
  • f32 vector dtype baseline
  • unified distance metric (cosine/dot/l2 per index)
  • consistent normalization policy
• Run dual-write and parity validation (SQLite vs external backend) before switching production traffic

License

MIT