> For the complete documentation index, see [llms.txt](https://docs.cherryai.com.cn/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cherryai.com.cn/docs/en-us/question-contact/knowledge.md). # Knowledge Primer ## What are tokens? Tokens are the basic units that AI models use to process text, and can be understood as the smallest unit of the model's "thinking." They are not exactly the same as the characters or words we understand, but rather a special way the model splits text. #### 1. Chinese tokenization * A single Chinese character is usually encoded as 1–2 tokens * For example:`"Hello"` ≈ 2–4 tokens #### 2. English tokenization * Common words are usually 1 token * Longer or less common words will be split into multiple tokens * For example: * `"hello"` = 1 token * `"indescribable"` = 4 tokens #### 3. Special characters * Spaces, punctuation marks, etc. also take up tokens * A newline is usually 1 token {% hint style="info" %} The tokenizers used by different providers are all different, and even models from the same provider may use slightly different tokenizers. This knowledge is only meant to clarify the concept of a token. {% endhint %} *** ## What is a tokenizer? A tokenizer is a tool that AI models use to convert text into tokens. It determines how input text is split into the smallest units the model can understand. ### Why are tokenizers different for different models? #### 1. Different training data * Different corpora lead to different optimization directions * Different levels of multilingual support * Specialized optimization for specific domains (medical, legal, etc.) #### 2. Different tokenization algorithms * BPE (Byte Pair Encoding) - OpenAI GPT series * WordPiece - Google BERT * SentencePiece - suitable for multilingual scenarios #### 3. Different optimization goals * Some focus on compression efficiency * Some focus on preserving semantics * Some focus on processing speed ### Practical impact The same text may have a different number of tokens in different models: ``` Input: "Hello, world!" GPT-3: 4 tokens BERT: 3 tokens Claude: 3 tokens ``` *** ## What is an embedding model? **Basic concept:** An embedding model is a technology that converts high-dimensional discrete data (text, images, etc.) into low-dimensional continuous vectors. This transformation allows machines to better understand and process complex data. Imagine simplifying a complex puzzle into a simple coordinate point, while still preserving the puzzle's key features. In the large-model ecosystem, it acts as a "translator," converting information understandable to humans into a numerical form that AI can compute. **How it works:** Taking natural language processing as an example, an embedding model can map words to specific positions in vector space. In this space, words with similar meanings naturally cluster together. For example: * The vectors for "king" and "queen" are very close * Pet-related words like "cat" and "dog" are also close to each other * Whereas words with unrelated meanings like "car" and "bread" are farther apart **Main application scenarios:** * Text analysis: document classification, sentiment analysis * Recommendation systems: personalized content recommendations * Image processing: similar image retrieval * Search engines: semantic search optimization **Core advantages:** 1. Dimensionality reduction: simplifies complex data into easily processed vector representations 2. Semantic preservation: retains key semantic information from the original data 3. Computational efficiency: significantly improves the training and inference efficiency of machine learning models **Technical value:** Embedding models are foundational components of modern AI systems. They provide high-quality data representations for machine learning tasks and are key technologies driving the development of fields such as natural language processing and computer vision. *** ## How Embedding models work in knowledge retrieval **Basic workflow:** 1. **Knowledge base preprocessing stage** * Split documents into appropriately sized chunks (text blocks) * Use an embedding model to convert each chunk into a vector * Store the vectors and original text in a vector database 2. **Query processing stage** * Convert the user's question into a vector * Retrieve similar content from the vector database * Provide the retrieved relevant content as context to the LLM *** ## **What is MCP (Model Context Protocol)?** MCP is an open-source protocol designed to provide context information to large language models (LLMs) in a standardized way. * **Analogy:** You can think of MCP as the AI world's "USB drive." We know that a USB drive can store all kinds of files, and once plugged into a computer, it can be used directly. Similarly, various "plugins" that provide context can be "plugged in" on an MCP Server, and an LLM can request these plugins from the MCP Server as needed, thereby obtaining richer contextual information and enhancing its own capabilities. * **Comparison with Function Tool:** Traditional Function Tools can also provide external capabilities to an LLM, but MCP is more like a higher-level abstraction. Function Tools are more task-specific tools, whereas MCP provides a more general, modular mechanism for obtaining context. ### **Core advantages of MCP** 1. **Standardization:** MCP provides a unified interface and data format, enabling different LLMs and context providers to work together seamlessly. 2. **Modularity:** MCP allows developers to break context information into independent modules (plugins), making them easier to manage and reuse. 3. **Flexibility:** LLMs can dynamically choose the context plugins they need based on their own requirements, enabling smarter and more personalized interactions. 4. **Scalability:** MCP's design supports adding more types of context plugins in the future, offering unlimited possibilities for expanding LLM capabilities. *** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.cherryai.com.cn/docs/en-us/question-contact/knowledge.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.