Knowledge Popularization
What are tokens?
Tokens are the basic units that AI models use to process text, and can be understood as the smallest unit of the model's "thinking." They are not exactly the same as the characters or words we understand, but rather a special way the model splits text.
1. Chinese tokenization
A single Chinese character is usually encoded as 1–2 tokens
For example:
"Hello"≈ 2–4 tokens
2. English tokenization
Common words are usually 1 token
Longer or less common words will be split into multiple tokens
For example:
"hello"= 1 token"indescribable"= 4 tokens
3. Special characters
Spaces, punctuation marks, etc. also take up tokens
A newline is usually 1 token
The tokenizers used by different providers are all different, and even models from the same provider may use slightly different tokenizers. This knowledge is only meant to clarify the concept of a token.
What is a tokenizer?
A tokenizer is a tool that AI models use to convert text into tokens. It determines how input text is split into the smallest units the model can understand.
Why are tokenizers different for different models?
1. Different training data
Different corpora lead to different optimization directions
Different levels of multilingual support
Specialized optimization for specific domains (medical, legal, etc.)
2. Different tokenization algorithms
BPE (Byte Pair Encoding) - OpenAI GPT series
WordPiece - Google BERT
SentencePiece - suitable for multilingual scenarios
3. Different optimization goals
Some focus on compression efficiency
Some focus on preserving semantics
Some focus on processing speed
Practical impact
The same text may have a different number of tokens in different models:
What is an embedding model?
Basic concept: An embedding model is a technology that converts high-dimensional discrete data (text, images, etc.) into low-dimensional continuous vectors. This transformation allows machines to better understand and process complex data. Imagine simplifying a complex puzzle into a simple coordinate point, while still preserving the puzzle's key features. In the large-model ecosystem, it acts as a "translator," converting information understandable to humans into a numerical form that AI can compute.
How it works: Taking natural language processing as an example, an embedding model can map words to specific positions in vector space. In this space, words with similar meanings naturally cluster together. For example:
The vectors for "king" and "queen" are very close
Pet-related words like "cat" and "dog" are also close to each other
Whereas words with unrelated meanings like "car" and "bread" are farther apart
Main application scenarios:
Text analysis: document classification, sentiment analysis
Recommendation systems: personalized content recommendations
Image processing: similar image retrieval
Search engines: semantic search optimization
Core advantages:
Dimensionality reduction: simplifies complex data into easily processed vector representations
Semantic preservation: retains key semantic information from the original data
Computational efficiency: significantly improves the training and inference efficiency of machine learning models
Technical value: Embedding models are foundational components of modern AI systems. They provide high-quality data representations for machine learning tasks and are key technologies driving the development of fields such as natural language processing and computer vision.
How Embedding models work in knowledge retrieval
Basic workflow:
Knowledge base preprocessing stage
Split documents into appropriately sized chunks (text blocks)
Use an embedding model to convert each chunk into a vector
Store the vectors and original text in a vector database
Query processing stage
Convert the user's question into a vector
Retrieve similar content from the vector database
Provide the retrieved relevant content as context to the LLM
What is MCP (Model Context Protocol)?
MCP is an open-source protocol designed to provide context information to large language models (LLMs) in a standardized way.
Analogy: You can think of MCP as the AI world's "USB drive." We know that a USB drive can store all kinds of files, and once plugged into a computer, it can be used directly. Similarly, various "plugins" that provide context can be "plugged in" on an MCP Server, and an LLM can request these plugins from the MCP Server as needed, thereby obtaining richer contextual information and enhancing its own capabilities.
Comparison with Function Tool: Traditional Function Tools can also provide external capabilities to an LLM, but MCP is more like a higher-level abstraction. Function Tools are more task-specific tools, whereas MCP provides a more general, modular mechanism for obtaining context.
Core advantages of MCP
Standardization: MCP provides a unified interface and data format, enabling different LLMs and context providers to work together seamlessly.
Modularity: MCP allows developers to break context information into independent modules (plugins), making them easier to manage and reuse.
Flexibility: LLMs can dynamically choose the context plugins they need based on their own requirements, enabling smarter and more personalized interactions.
Scalability: MCP's design supports adding more types of context plugins in the future, offering unlimited possibilities for expanding LLM capabilities.
Last updated
Was this helpful?