Chinese

The chinese analyzer is designed specifically to handle Chinese text, providing effective segmentation and tokenization.

Definition

The chinese analyzer consists of:

• Tokenizer: Uses the jieba tokenizer to segment Chinese text into tokens based on vocabulary and context. For more information, refer to Jieba.

• Filter: Uses the cnalphanumonly filter to remove tokens that contain any non-Chinese characters. For more information, refer to Cnalphanumonly.

The functionality of the chinese analyzer is equivalent to the following custom analyzer configuration:

analyzer_params = {
    "tokenizer": "jieba",
    "filter": ["cnalphanumonly"]
}

Configuration

To apply the chinese analyzer to a field, simply set type to chinese in analyzer_params.

analyzer_params = {
    "type": "chinese",
}

Example output

Here’s how the chinese analyzer processes text.

Original text:

"Milvus 是一个高性能、可扩展的向量数据库！"

Expected output:

["Milvus", "是", "一个", "高性", "性能", "高性能", "可", "扩展", "的", "向量", "数据", "据库", "数据库"]