Build a Local RAG Pipeline with LanceDB and Small LLMs

What a Local RAG Pipeline Is and Why You’d Build One

A local RAG pipeline is a document processing AI system that stores your own files as embeddings in a vector database and uses a small language model to answer questions and summarize them entirely on your machine, without sending content to external services. It is worth building if you have many PDFs, notes, or images that you want to search and query in natural language and you care about low latency and predictable costs. The catch: you need enough hardware to run a small language model and a bit of comfort with Python, but you do not need giant GPUs or cloud infrastructure. Instead of chasing the largest models, you focus on a tight combination of an efficient vector database setup and a focused small language model so your own documents become instantly queryable.

Under the hood, the “RAG” part—Retrieval-Augmented Generation—means you first store document chunks and their embeddings, then feed the most relevant chunks into the model each time you ask a question. A vector database keeps similar chunks close together, so similarity search turns your question into a short list of likely answers. LanceDB is an open-source vector database built for this exact job, with native support for text, vectors, images, audio, and video in the same table. On the model side, modern small language models at the 3B scale, such as SmolLM3, are trained so well that for focused, domain-specific tasks they match or beat much larger 70B models while running on a single consumer GPU at a fraction of the operating cost.

Prerequisites: Hardware, Keys, and Python Environment

Before wiring up a local RAG pipeline, confirm you can meet the hardware minimums and basic software requirements. For GPU-backed inference, you should have at least 6 GB of VRAM in bfloat16, with 8 GB or more on something like an RTX 3060 recommended; on the system side, aim for 16 GB RAM minimum and 32 GB as a comfortable target, plus around 8 GB of free disk space (20 GB+ SSD recommended). Apple Silicon machines work as well, starting around an M2 with 8 GB and scaling up to an M2 Pro or M3 with 16 GB. CPU-only setups also work, though you will see slower generation speeds. The other caveat: you need an embedding provider. A common route is an OpenAI key for creating embeddings, although any compatible alternative is acceptable.

Set up a Python environment once these basics are ready. Use Python 3.10 or newer and create a virtual environment so your RAG pipeline’s dependencies stay isolated from other projects. A helpful detail from the LanceDB installation notes: uv is recommended for faster installation of packages if you use it in place of standard pip. Inside the environment, you will install three main groups of tools: the LanceDB stack and PDF utilities, the small language model stack based on the transformers library, and any client libraries you need for your embedding provider. Doing this up front saves you from debugging missing modules later, which is one of the most annoying gotchas when assembling a new AI pipeline.

Step-by-Step: Wiring LanceDB and a Small LLM into a Local RAG Pipeline

1. Create and activate a Python 3.10+ virtual environment, then install LanceDB, PDF utilities, and AI packages using uv or pip.
2. Initialize a local LanceDB database connection and inspect existing tables to confirm the vector database setup.
3. Ingest a PDF, chunk it into pages or sections, and create embeddings for each chunk using your embedding provider.
4. Store chunks, embeddings, and any metadata in a LanceDB table configured with an appropriate ANN index.
5. Detect your device type (CUDA GPU, Apple Silicon, or CPU) and load a small language model like SmolLM3 through transformers.
6. Implement a retrieval function that queries LanceDB for the most similar chunks to a user query and returns them as context.
7. Wrap the retrieval and generation into a single RAG function that answers questions or summarizes documents by passing retrieved chunks into the small LLM.

Let’s walk through those steps in more detail so they feel like a build, not a checklist. Start with installations: inside your virtual environment, install LanceDB and friends with a single command such as “uv pip install lancedb pandas pyarrow pypdf pillow numpy openai open-clip-torch torch”. This pulls in the vector database, data frames, Arrow support, PDF handling, image tools, numerical libraries, and a client for OpenAI embeddings. Next, initialize LanceDB locally: connect to a path like “./lancedb_data” and confirm that the connection works by printing the existing table names. This is the core of your local RAG pipeline—everything stays on disk under that directory. The main gotcha here is path management: choose a stable location so you do not accidentally reconnect to an empty folder and think your data vanished.

Once LanceDB is up, feed it documents. For example, ingest a PDF via pathlib, pointing to a file like "assets/exploring-ann-algorithms.pdf" and asserting that the path exists so you fail fast if the file is missing. Chunk the PDF into pages so each page becomes a separate text piece; one example pipeline chunks that particular PDF into 14 parts (each page being a chunk) before embedding them into the vector table. Store these chunks together with their embeddings and metadata in a LanceDB table. By indexing these vectors with approximate nearest neighbor (ANN) algorithms like IVF or HNSW, you trade a small amount of recall for significant speedups when searching over large collections. Done right, queries such as "find the page explaining how HNSW works" return the most similar items first, and you can visually confirm that, for image data, the peacock image query returns the peacock image as the closest match.

Now focus on the language model. Use the transformers library and related packages—torch, accelerate, bitsandbytes, sentencepiece, trl, peft, and datasets—to load a small language model like SmolLM3. Run a device detection helper to see whether a CUDA GPU is available; a typical expected output on a suitable machine would note "CUDA GPU detected" and list details such as an NVIDIA GeForce RTX 3060 with its VRAM amount, plus a device string like "cuda" and dtype settings such as torch.bfloat16. The benefit of a 3B-scale model here is that it fits entirely on that consumer GPU, loads in seconds, and carries no per-token cost beyond your hardware. According to the description of SmolLM3’s training, it was trained on 11.2 trillion tokens with a staged curriculum and 140 billion reasoning tokens in post-training, which is why it can rival larger 4B models on several benchmarks.

Finally, tie retrieval and generation together. Implement a function that takes a user query, embeds it, and runs a similarity search against the LanceDB table. Because the database keeps high-dimensional embeddings indexed so similar vectors sit near each other, your query returns the most relevant document chunks for that question. Feed those chunks into the small language model as context, and ask it to answer or summarize. This is your RAG function: the vector database finds the right bits of your documents, and the model turns them into natural language output. On focused, domain-specific tasks like this, a well-tuned small language model will match or beat a 70B model while using a tenth of the operating cost. The main gotcha at this stage is prompt design—make sure you clearly separate retrieved context from user questions so the model does not hallucinate outside your documents.

Why LanceDB and Small Language Models Fit Local Document Processing

LanceDB’s design makes it a natural fit for a local RAG pipeline. It is open-source and can be used as an embedded vector database on your machine or self-hosted if you need a server setup. Because it is multimodal by design, text, vectors, images, audio, and video can live as columns in the same table instead of being split across separate structures. That simplifies your document processing AI when you have mixed content—think reports with figures or audio transcripts linked to slides. On top of that, LanceDB supports multiple index types such as IVF, HNSW, PQ, and RQ for vectors, plus BM25 for full-text search, which gives you hybrid search out of the box. You can store chunks, metadata, and media in a single versioned, embedded table with ANN indexing and object storage support bundled in.

On the modeling side, modern small language models make the pipeline responsive and affordable. The Hugging Face small language model collection includes SmolLM3-3B (instruction-tuned), SmolLM3-3B-Base (untuned), SmolLM2-1.7B, and SmolVLM for vision-language tasks. SmolLM3 in particular offers dual-mode reasoning—"think" and "no_think"—within a single checkpoint, a 128k context window, and native tool calling. The architecture includes grouped query attention to reduce key-value cache memory by around 25%, and a 3:1 RoPE-to-NoPE ratio to generalize better over long contexts. For everything focused and domain-specific, the small language model fine-tuned on your data will match a larger model at a tenth of the operating cost. That combination of LanceDB’s efficient retrieval and a small LLM’s strengths is what makes a local RAG pipeline feel fast and tailored to your own documents.

Takeaway: A Fast, Focused Local RAG You Can Grow Over Time

Once you follow the steps above—set up the environment, initialize LanceDB, ingest and embed your documents, and connect a small language model through transformers—you end up with a practical local RAG pipeline that searches, retrieves, and summarizes your own PDFs and other media. You can see it working when similarity queries bring back the expected results, such as an HNSW-focused page for an ANN question or a peacock image for an image search. From here, you can extend the pipeline: add hybrid search with BM25, store audio transcripts alongside text, or fine-tune the small language model on your company’s writing style. The real thing to watch for is scope creep—start with one or two document types and a clear use case, such as internal report search, and let the system prove its value before you widen it. Once it is stable, you will have a reliable, fast document processing AI that feels like an always-on assistant for your own files.