Doco

Your AI Agent Does Not Need RAG. It Needs a Readable Knowledge Base.

Your AI Agent Does Not Need RAG. It Needs a Readable Knowledge Base.

An AI agent does not automatically need a custom RAG pipeline to use organizational knowledge. If the corpus is maintained, searchable, structurally readable, citable, and available through bounded tools, direct retrieval can be the simpler system. Use RAG when ranking and synthesizing across a large or heterogeneous corpus is genuinely the problem.

Doco series · Article 12 · Architecture opinion

“We need the agent to answer from our documents” often becomes “we need RAG” before anyone writes down the actual job.

That jump confuses a technique with a requirement. Retrieval-augmented generation is a valuable family of architectures. The original RAG paper by Lewis et al. combines a generator's parametric memory with retrieved non-parametric memory for knowledge-intensive tasks. It is especially relevant when the system must rank evidence from a large corpus and generate an answer from it.

But many workplace agents are not open-domain question-answering systems. They need to find a policy, inspect the relevant section, cite it, and sometimes update one paragraph. For that workflow, the first problem is often not model training, embeddings, or chunk orchestration. It is that the knowledge base is difficult for software to read safely.

Start with the job, not the acronym

Consider four requests:

  1. “What is the current refund window? Show me the policy.”

  2. “Update the on-call owner in the release checklist.”

  3. “Compare every customer interview about onboarding confusion.”

  4. “Answer arbitrary questions across ten million mixed-format documents.”

The first request needs search, a canonical source, and a citation. The second needs structured mutation and concurrency protection. The third may benefit from ranking, clustering, or semantic retrieval. The fourth is a strong candidate for a serious retrieval pipeline.

Calling all four “RAG” hides the differences that determine the architecture.

What a readable knowledge base exposes

Readable does not mean “the model can receive a giant Markdown export.” It means the system provides a deliberate set of operations:

  • Browse: list knowledge bases, folders, documents, and outlines.

  • Search: find relevant blocks with paths and surrounding context.

  • Read: request Markdown, plain text, or lossless structured content.

  • Budget: limit output and continue from an explicit cursor.

  • Cite: return a stable document and block address.

  • Check freshness: identify the source and index versions.

  • Edit safely: patch a bounded unit against the version that was read.

These operations let the agent progressively disclose context. It can inspect an outline before loading a section, and search before reading a whole document. The result is not magical retrieval. It is a knowledge interface whose behavior can be tested.

Doco is the system I am building, so this is a first-party architecture argument. Doco currently uses structured full-text search rather than claiming universal semantic search. SQLite's official FTS5 documentation describes a full-text search virtual table with phrase, prefix, NEAR, and boolean query support. That kind of retrieval is often enough for known policies, identifiers, names, and operational terms.

Chunking is not neutral

A typical RAG pipeline copies source documents, divides them into chunks, embeds those chunks, and builds a retrieval index. Each step introduces a projection that can drift from the source:

  • a heading may be separated from the paragraph it qualifies;

  • a table row may lose its headers;

  • a chunk ID may change after a small source edit;

  • the index may lag behind the document;

  • a generated answer may cite a chunk that has no durable user-facing address.

Those are solvable engineering problems. They are still problems. If the source system already has stable blocks, heading paths, versions, and search, preserve those semantics before inventing another identity layer.

Direct retrieval is not “stuff everything into context”

The alternative to RAG is not a 200,000-token dump. A tool-using agent can follow a staged loop:

list → inspect outline → search → read relevant blocks → verify source → answer

The Model Context Protocol separates resources and tools so a server can expose contextual data and executable operations through a negotiated interface; see the MCP server concepts. That makes direct, structured retrieval portable across agent clients without requiring the knowledge base to pretend it is a chat model.

The agent should also know when the result is incomplete. “No match in the first page” must not become “the knowledge base contains no answer.” Search cursors, completeness flags, and index freshness are mundane metadata with large reliability consequences.

When RAG is the right choice

Use RAG—or another dedicated retrieval architecture—when one or more of these conditions dominate:

  • the corpus is too large for deterministic browsing and lexical search alone;

  • users ask broad natural-language questions with weak keyword overlap;

  • evidence comes from heterogeneous external sources without shared structure;

  • ranking quality is a core product capability worth evaluating continuously;

  • the system must synthesize several passages for every answer;

  • you can operate ingestion, deduplication, access control, freshness, and retrieval evaluation as real production systems.

Even then, RAG does not replace a readable source of truth. It becomes a projection over that source. Every retrieved passage should retain enough provenance to reach the authoritative document.

When direct knowledge tools are enough

A simpler tool layer is often sufficient when:

  • the corpus is a bounded team workspace;

  • documents have useful titles, headings, and stable structure;

  • queries contain operational vocabulary found in the source;

  • the agent must cite or edit the exact source;

  • freshness matters more than fuzzy recall;

  • the team values debuggable behavior over opaque ranking.

For example, an incident assistant looking up “payments rollback timeout” may perform better with full-text search over current runbooks than with a complex semantic stack that was indexed yesterday. The important comparison is empirical: measure whether the correct source block is found, whether its version is current, and whether the answer cites it.

The architecture can grow incrementally

Starting with a readable knowledge base does not prevent semantic retrieval later. It creates a better foundation for it.

  1. Make canonical documents structurally accessible.

  2. Add stable block IDs and source versions.

  3. Implement deterministic search with complete, testable results.

  4. Record real queries that deterministic retrieval misses.

  5. Add semantic ranking where those failures justify it.

  6. Keep citations bound to the canonical blocks, not only vector-store chunks.

This sequence turns RAG from a fashionable default into a targeted response to measured retrieval failures.

FAQ

Is this an argument against RAG?

No. RAG is useful for knowledge-intensive generation, especially across large or heterogeneous corpora. The argument is against treating it as the automatic first requirement for every agent that needs documents.

Is full-text search enough for every knowledge base?

No. It can miss paraphrases and concepts that share little vocabulary. It is strong for names, identifiers, policies, and operational phrases. Evaluate it against real queries before adding or rejecting semantic retrieval.

Can an agent use both direct tools and RAG?

Yes. A semantic retriever can identify candidate blocks, while direct tools fetch current canonical content, versions, and surrounding structure. The two layers should not create competing sources of truth.

What should I measure before building RAG?

Measure source-block recall, citation correctness, freshness, incomplete-search handling, latency, and the share of real queries that simpler retrieval cannot answer. Those results reveal whether the bottleneck is retrieval, document quality, or workflow design.

Bottom line

Your agent needs a dependable way to find, read, cite, and sometimes update knowledge. RAG may be part of that system, but it is not the definition of it. Build a readable source of truth first; add specialized retrieval when evidence shows that you need it.