Back to Blogs

How to Build a RAG Chatbot That Cites Your Company Data

12 min read

To build a RAG chatbot that cites your company data, retrieve relevant document passages, give those passages to an LLM, and attach each answer’s claims to the sources that support them. The critical step is preserving source information throughout the pipeline—not asking the model to invent references afterward.

This guide covers document preparation, retrieval, citation validation, access control, and testing.

What Is a RAG Chatbot?

A RAG chatbot combines retrieval-augmented generation with a conversational interface. Instead of answering only from a model’s training data, it searches an external knowledge base and uses the retrieved material to construct its response.

For company use, that knowledge base might contain:

  • Product documentation and release notes.
  • Internal operating procedures.
  • Customer support articles.
  • Employee onboarding guides.
  • Approved technical specifications.

The basic workflow is:

Question → authorized document retrieval → evidence-based answer → verified citations.

RAG does not automatically make an answer accurate. The chatbot can retrieve irrelevant passages, misinterpret a source, or attach a valid citation to an unsupported statement.

A citation is useful only when its source supports the claim beside it.

Before You Start

Choose a narrow use case before connecting every company system. A chatbot answering product-support questions from approved documentation is easier to test than an assistant searching an entire organization.

You need:

  • A defined audience: customers, employees, or a specific internal team.
  • An approved document collection: with owners, versions, and access rules.
  • A retrieval system and an LLM: connected through a backend you control.

Decide what the chatbot must not answer. Questions requiring an account-specific lookup, a live inventory check, or an operational action need separate systems; a document search alone cannot establish those facts.

The non-obvious requirement is source traceability. Preserve document identifiers, locations, and permissions before chunking. Reconstructing that information after indexing creates avoidable citation and security problems.

Prepare Your Company Data

1. Select Authoritative Sources

Start with documents that someone owns and maintains. Exclude abandoned drafts, duplicated exports, and material whose approval status is unclear.

For each document, record:

  • Document identifier and title.
  • Canonical source location.
  • Owner and approval status.
  • Version and last-updated timestamp.
  • Access permissions.
  • Whether it supersedes another document.

Do not assume the most recently uploaded file is authoritative. A newer draft should not override an approved procedure.

For conflicting sources, establish a precedence rule. For example, an approved product specification takes priority over an informal discussion about that product.

Expected result: every indexed document has an identifiable owner, status, and source location.

2. Extract Text Without Losing Its Structure

Document extraction should preserve headings, paragraphs, lists, and table relationships.

A table containing a feature name, plan eligibility, and an exception becomes misleading if those cells are separated. Likewise, a procedure can lose its meaning when a warning is detached from the steps it qualifies.

For scanned documents, use OCR and review representative output before indexing. Check product names, punctuation, and table cells rather than accepting readable-looking text as accurate.

Microsoft’s documentation on RAG and document extraction describes semantic chunking around document structure. The practical lesson is straightforward: preserve meaningful sections rather than treating every file as an undifferentiated string.

Expected result: extracted text retains the context needed to interpret each passage.

3. Create Chunks With Citation Metadata

A chunk is a searchable passage. It should contain enough context to answer a question without including so much unrelated material that the evidence becomes difficult to isolate.

Split around headings and paragraph boundaries first. Keep procedures, definitions, and small tables together where practical.

There is no universal chunk size that works for every document collection. Test your segmentation against the questions users actually ask.

Each chunk should retain:

  • A stable chunk identifier.
  • Its parent document identifier.
  • Document title and version.
  • Section heading.
  • Page number or another source location.
  • Canonical source link, when available.
  • Permission metadata.

For a web page, a section anchor can provide a useful citation target. For a PDF, preserve the original page mapping. If no reliable page number exists, cite a section or passage instead of inventing one.

Expected result: every retrieved passage can be traced back to a specific source location.

Build Retrieval Around Real Questions

4. Index Content and Metadata Together

Embeddings represent text as vectors so your retrieval system can search for related meaning. Store the vector alongside the original chunk text and its metadata.

Keep the indexing and query embedding configurations compatible. A change to the embedding model can require re-embedding stored content; treat that as an index migration, not a casual setting change.

Also preserve searchable text. Company questions often contain exact identifiers that matter:

  • Product codes.
  • Error messages.
  • Feature names.
  • Acronyms.
  • Document titles.

Vector search handles semantic similarity. Keyword search handles exact terms. Hybrid retrieval combines both, making it a useful design to evaluate when your corpus includes technical language and precise identifiers.

Expected result: the retrieval system can return both meaningful matches and exact-term matches.

5. Enforce Permissions Before Returning Evidence

Authenticate the user before searching restricted company data. The backend should derive their permissions from a trusted identity system, not from a role written in the chat message.

Apply those permissions during retrieval. Unauthorized chunks must not reach the model.

The same rule applies to reranking: if another service scores candidate passages, send it only material the user is allowed to access.

Microsoft’s RAG guidance explicitly recommends access control at retrieval time. This is an application boundary, not a prompt instruction.

Also protect citations themselves. A restricted title, excerpt, or document link can disclose information even when the answer omits the underlying content.

Expected result: the chatbot searches and cites only documents the current user can access.

6. Rank Passages and Check Evidence Coverage

Retrieval should identify passages that answer the question, not merely passages that discuss the same topic.

A reranker can reorder authorized candidates according to their relevance. After ranking, check whether the selected passages contain the information required for an answer.

For example, documentation mentioning “data export” does not necessarily answer whether export includes attachments. The chatbot needs a passage addressing that specific condition.

Do not treat a similarity score as a calibrated probability of correctness. Its meaning depends on the retrieval method, model, and corpus.

When evidence is insufficient, return a bounded response:

“I couldn’t find an approved source that confirms whether attachments are included.”

Expected result: the answer receives relevant evidence, or the chatbot clearly states what it cannot establish.

Generate Answers With Verifiable Citations

7. Give the Model Explicit Evidence Labels

Assign temporary labels to the retrieved passages, such as S1, S2, and S3. Each label should map to a source record maintained by your backend.

Send the passage text and relevant context to the model. Keep trusted instructions separate from retrieved document content.

A useful generation instruction is:

“Answer company-specific questions only from the supplied evidence. Cite each factual claim using the supplied source labels. Do not invent source labels, document titles, links, or page numbers. If the evidence is missing or conflicting, say so.”

Add a separate instruction that retrieved passages are reference material, not commands.

This helps define the model’s task, but it is not a security guarantee. A document can contain malicious instructions; the application still needs permission controls, output validation, and restrictions on any connected tools.

Expected result: the model produces claims tied to evidence labels rather than fabricated references.

8. Resolve Citations in the Backend

Let the model select source labels. Let the backend construct the actual citation links.

This keeps titles, locations, and URLs under application control.

A simplified workflow might look like this:

user = authenticate(request) access_rules = resolve_permissions(user) question = interpret_follow_up(request, conversation)

candidates = retrieve(question, access_rules) evidence = rank_authorized_passages(question, candidates) source_map = assign_source_labels(evidence)

draft = generate_grounded_answer(question, source_map) result = validate_claims_and_citations(draft, source_map)

return render_authorized_answer(result, source_map)

This is conceptual pseudocode, not a provider-specific implementation. Its key property is that permissions and source records remain outside the model’s control.

In the interface, place citations beside the claims they support. A source list at the bottom of a long answer makes the relationship harder to inspect.

A useful citation shows the document title, section or page, and a way to inspect the supporting passage.

Expected result: citation links resolve to known, authorized source records.

9. Validate More Than Citation Existence

A citation can be real and still be wrong.

Use separate checks for:

  • Existence: does the source label appear in the supplied evidence?
  • Support: does the cited passage substantiate the claim?
  • Coverage: are company-specific factual claims supported?
  • Access: is the source still authorized for this user?

If validation fails, revise the answer, remove the unsupported claim, or decline to answer. Do not simply attach another citation to make the output look complete.

For illustration, imagine a fictional support document states: “Bulk export supports CSV files.” That passage does not support the claim that bulk export also includes images.

An automated support checker can help detect such mismatches, but it can also make mistakes. Review failures and high-consequence answers with humans.

Expected result: the chatbot checks whether evidence supports the answer—not just whether reference labels are formatted correctly.

Keep the Knowledge Base Current

An accurate answer can become wrong when its source becomes obsolete.

Track document updates, deletions, and permission changes. When content changes, update the affected chunks and embeddings. When a document is removed or access is revoked, remove or restrict the corresponding indexed content.

Caches need the same treatment. A cached answer should not expose a document that the user can no longer access.

Where practical, record which document versions supported an answer. That makes investigation easier when someone reports an outdated response.

If your chatbot combines document retrieval with live system data, label the two separately. An approved guide can explain how an order status works; an authenticated system lookup establishes the status of a particular order.

Test the Chatbot Before Expanding It

Build an evaluation set from real user questions and have subject-matter experts mark the expected evidence.

Include these test categories:

  • Questions answered by one passage.
  • Questions requiring several sources.
  • Questions with no documented answer.
  • Conflicting or superseded documents.
  • Restricted documents requested by unauthorized users.
  • Exact identifiers and ambiguous follow-up questions.
  • Documents containing instructions that should be ignored.

Measure retrieval and answer quality separately:

  • Evidence retrieval: did the system find the required passage?
  • Answer support: are the answer’s claims established by the evidence?
  • Citation accuracy: do citations point to supporting passages?
  • Abstention quality: does the chatbot decline when evidence is insufficient?
  • Access enforcement: does it exclude restricted material?
  • Response time: is the experience acceptable for the intended workflow?

Separate measurements tell you where to fix the system. Missing evidence is a retrieval problem; unsupported claims despite correct evidence are a generation or validation problem.

Rerun the evaluation set after changes to parsing, chunking, embeddings, ranking, or prompts.

Common Problems and Their Fixes

The Chatbot Finds the Right Document but Gives the Wrong Answer

Inspect the retrieved passage. It may omit the exception, prerequisite, or table header needed to interpret the answer.

Fix the extraction or chunk boundaries before changing the model.

Citations Point to Documents but Not Supporting Passages

Preserve section and page metadata, then validate claims against the actual cited text. Document-level relevance alone is not enough.

Old Documents Override Current Guidance

Filter out superseded material and make approval status part of retrieval. Use document precedence rules rather than relying on upload dates.

Follow-Up Questions Retrieve Irrelevant Evidence

Use conversation context to rewrite the follow-up into a standalone search query. Do not treat earlier model answers as authoritative company evidence.

The Chatbot Answers When Nothing Supports Its Response

Define an evidence-insufficient response and test it deliberately. An honest refusal is better than a fluent answer with decorative citations.

FAQ

What Is the Difference Between a RAG Chatbot and a Regular Chatbot?

A RAG chatbot retrieves external information before generating an answer. A chatbot without retrieval relies on its model knowledge and conversation context, which do not establish current company-specific facts.

Do I Need to Fine-Tune a Model on My Company Documents?

No. A RAG chatbot can retrieve company documents at answer time without fine-tuning. Fine-tuning serves different purposes, such as shaping behavior or output style; it does not replace current evidence retrieval.

Can a RAG Chatbot Still Hallucinate?

Yes. It can misread retrieved text, introduce unsupported details, or cite a passage that does not support its claim. Evidence-based prompting, citation validation, and abstention reduce the risk but do not eliminate it.

What Should a RAG Chatbot Citation Include?

Include the source title and a precise location, such as a section, page, or passage. Resolve the citation from stored metadata and let authorized users inspect the supporting evidence.

How Do I Prevent Employees From Seeing Restricted Documents?

Authenticate users and apply document permissions during retrieval. Enforce access on excerpts, citations, caches, and source links as well; a prompt telling the model to hide confidential data is insufficient.

Can a RAG Chatbot Read Scanned PDFs?

Yes, provided the ingestion pipeline extracts their text with OCR. Review extraction quality and preserve page mappings so citations remain useful and accurate.

What Should the Chatbot Do When Company Sources Disagree?

It should follow an established authority or version rule when one resolves the conflict. Otherwise, it should explain the disagreement, cite the conflicting sources, and avoid presenting either claim as settled.

What Is the Best First Version to Build?

Build a read-only chatbot over a narrow, approved document collection. Prove that it retrieves the right evidence, cites accurately, respects permissions, and declines unsupported questions before expanding its scope.

Back to Blogs