Check materials and recall

Recall testing directly checks whether the “right fragment can be found” and does not go through chat model polishing first. It helps you determine whether the issue lies in the data, parsing, chunking, or retrieval settings.

Prepare 3–5 real questions whose answers you already know, and reuse them after every update to the data, model, or chunking. Fixed questions make regressions easier to spot than ad hoc test questions.

Prepare test questions

It is recommended to cover three types of questions at the same time:

• Exact facts, for example, “What is the accommodation cap in first-tier domestic cities?”
• Conditional rules, for example, “In what situations can overseas car rentals be reimbursed?”
• Easy-to-confuse cases, for example, “Who must give additional approval for business trips over 5,000 yuan?”

Do not test only with document titles or full original sentences, as that will overestimate real-world effectiveness.

Complete a recall test

1

1. Open Recall Test

Open 【Knowledge Base】 in the left navigation → select a knowledge base → enter 【Recall Test】.

2

2. Enter a real question

Enter a question with a clear answer and run the test. The question should be close to everyday wording and should not deliberately copy the original document text.

3

3. Check the source and fragment

Confirm that the source is correct and that the fragment contains both the conditions and the conclusion needed to answer the question.

Do not just look at whether there is a result; also check the source, fragment completeness, and order.

4

4. Diagnose the problem by comparing symptoms

If there is no correct fragment at all, first check the data, parsing, and chunking; if the correct fragment appears but the order is unstable, then consider embeddings, reranking, or Top K.

5

5. Adjust one item and retest

Change only one setting per round. If parsing, chunking, or indexing is involved, run 【Reindex】 first, then retest with the same set of questions.

How to read the results

Symptom	Explanation	Next step
Correct source appears first, fragment is complete	Recall is basically acceptable	Test a few more phrasings
No correct result at all	The data is not ready, content is missing, the phrasing differs too much, or the threshold is too high	Check the data, main text, Chunks, and retrieval settings in order
Source is correct but the fragment is missing key sentences	Parsing or chunk boundary is not ideal	View Chunks, adjust, and reindex
Old and new versions appear at the same time	All documents with the same name were retained	Delete the old entry or re-import using 【Replace】
Correct items often appear later	Too many candidates or unstable ranking	Clean up the data, and consider an embedding or reranking model
Recall is correct but the chat answer is inaccurate	The problem is more likely in the prompt or chat model	Keep the recall settings and adjust the question prompt and chat model

The chat model cannot make up for key information that was not found during recall. When recall results are poor, do not first try repeatedly swapping chat models to troubleshoot.

Tuning loop

Fixed questions → check results → identify the layer → adjust one item → reindex if needed → retest.

Recommended order:

1. Confirm that the data is correct and there are no duplicates or outdated versions.
2. Check the parsed main text and Chunks.
3. When the phrasing differs greatly from the original text, consider the embedding model.
4. When the candidate is roughly correct but the order is unstable, then consider the reranking model.
5. Reindex after adjusting, and repeat the same set of tests.

If fragments are incomplete, check the chunking settings again; changes only affect new data, and old data needs to be reindexed.

Configuration notes

Configuration item	Product default	Suggested starting point	Effect	Applicable scenario	Notes
Number of test questions	—	3–5	Establish a repeatable quality baseline	All knowledge bases	Cover exact facts, conditional rules, and easily confused questions
Top K	6, optional 1–50	Keep 6 for now	Control the number of final fragments	Trade-off between coverage and noise	Increasing it may consume more context
Similarity threshold	0.0, shown only when reranking is configured	Start from 0.0	Filter out low-scoring results after reranking	Noise still remains after reranking	Setting it too high will remove correct fragments
Retest method	—	Change only one item each round	Determine where the setting change comes from	Tuning, updating data, or model	After modifying chunking or the model, reindex first

Expected result

• The correct source appears stably among the first few results.
• The fragment contains the conditions and conclusion needed to answer the question.
• After rephrasing it naturally, the result remains stable.
• After updating the data or settings, the fixed questions show no obvious degradation.

User case

Xiaolin found that the “accommodation expense standard” could be matched using the original wording, but “how much can be reimbursed for a hotel stay at most” was unstable. He first confirmed that the data and Chunks were normal, then configured the embedding model and retested. After the correct fragment appeared, it sometimes ranked lower, so he then added a reranking model.

The completion criterion is: all three different phrasings should find the same accommodation standard among the first few results, and the fragment should include the applicable city and the maximum amount.

Frequently Asked Questions

If there is no correct fragment at all, should I first increase Top K?

First check the data main text and Chunks. If parsing or splitting is wrong, increasing Top K will only return more incorrect or incomplete fragments.

Why can't I see the similarity threshold?

Only after selecting a reranking model will 【Similarity Threshold】 be shown in the knowledge base settings.

What if recall is correct but the chat answer is still inaccurate?

Keep the current recall settings and check the question wording, conversation context, and chat model. At this point, the problem is usually no longer in the data retrieval layer.

Continue reading

Model and retrieval settings

Adjust embeddings, reranking, Top K, and chunking.

Document parsing and OCR

Handle missing text, garbled characters, and scanned content.

Use in conversations

After recall is qualified, use the knowledge base for questioning.