Private Knowledge Base Q&A

The HR team wants colleagues to look up current policies, but does not want the agent to fill in answers with common sense. This case combines the scope of materials, retrieval testing, and response rules; when content cannot be found, the agent should state it clearly.

Build a separate knowledge base for a single topic to avoid mixing current materials with outdated materials.

Adjust the RAG settings according to the structure of the materials only when retrieval results are unstable.

1) Test with real questions; 2) check the returned materials, original text snippets, and relevance. Only when these are stable, hand the knowledge base over to the agent.

Workflow

1

1. Build a knowledge base with a single topic

Put current materials together, remove duplicate and outdated versions. Make file names clearly state the topic and effective scope.

2

2. Use real questions for retrieval testing

At minimum, test easily confused clauses, colloquial questions, and questions with no answer. Adjust materials or RAG settings until the returned snippets are stable.

3

3. Create a dedicated agent

Bind only this knowledge base, and in the prompt require: “Prefer citing materials; say not found when you can’t find it; do not use common sense to fill in policies.” Keep permissions to the minimum scope.

4

4. Start with a small scope

First use it for internal lookup, not as a direct replacement for approvals or professional judgment. Collect incorrectly answered questions and go back to the materials and retrieval tests to correct them.

Example question rules

When answering, give the conclusion first, then specify the material name and relevant paragraph. When materials conflict, list the conflicts and do not choose for us. When the knowledge base has no answer, clearly write “Not found in current materials.”

Recommended combination and completion criteria

Item	Recommended approach
Knowledge base	One topic, one set of materials; delete duplicates and outdated versions
Retrieval test	Test standard phrasing, colloquial phrasing, and questions with no answer at the same time
Agent	Bind only the required knowledge base, and require citing materials without filling in missing content
Completion criteria	Can find the correct paragraph; conflicting materials will be listed; when there is no answer, clearly say not found

Content in the private knowledge base will be processed by the embedding and chat model services you choose. Before use, confirm the confidentiality requirements and service scope of the materials, and do not import keys and account credentials as knowledge materials.