停止词
stop 过滤器会从标记化文本中移除指定的停止词，帮助剔除常见的、意义不大的词。您可以使用stop_words 参数配置停用词列表。
配置
stop 过滤器可以通过stop_words 参数接受内联的停用词列表，也可以通过stop_words_file 参数从注册的文件资源中接受停用词列表。
内联 stop-words 列表
要在stop 过滤器中使用内联列表，请在过滤器配置中指定"type": "stop" 以及提供停用词列表的stop_words 参数。
analyzer_params = {
"tokenizer": "standard",
"filter":[{
"type": "stop", # Specifies the filter type as stop
"stop_words": ["of", "to", "_english_"], # Defines custom stop words and includes the English stop word list
}],
}
Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
Collections.singletonList(
new HashMap<String, Object>() {{
put("type", "stop");
put("stop_words", Arrays.asList("of", "to", "_english_"));
}}
)
);
const analyzer_params = {
"tokenizer": "standard",
"filter":[{
"type": "stop", # Specifies the filter type as stop
"stop_words": ["of", "to", "_english_"], # Defines custom stop words and includes the English stop word list
}],
};
analyzerParams = map[string]any{"tokenizer": "standard",
"filter": []any{map[string]any{
"type": "stop",
"stop_words": []string{"of", "to", "_english_"},
}}}
# restful
analyzerParams='{
"tokenizer": "standard",
"filter": [
{
"type": "stop",
"stop_words": [
"of",
"to",
"_english_"
]
}
]
}'
stop 过滤器接受以下可配置参数。
参数 |
说明 |
|---|---|
|
要从标记化中删除的单词列表。默认情况下，过滤器使用内置的
|
stop 过滤器对标记化器生成的术语进行操作，因此必须与标记化器结合使用。有关 Milvus 中可用的标记符列表，请参阅标准标记符及其同类页面。
定义analyzer_params 后，可以在定义 Collections Schema 时将其应用到VARCHAR 字段。这样，Milvus 就可以使用指定的分析器对该字段中的文本进行处理，从而实现高效的标记化和过滤。有关详情，请参阅示例使用。
从文件资源加载停止词Compatible with Milvus 3.0.x
对于大型自定义止语列表（特定语言列表、域词汇表或希望在多个 Collections 中共享的列表），可将止语存储在文件中，并将该文件注册为远程文件资源，然后通过stop_words_file 参数从过滤器中引用该文件。您可以单独使用stop_words_file ，也可以与内联stop_words 同时使用；当同时设置这两个参数时，过滤器会将这两个来源合并为一个单一的停止词语列表。
文件是纯 UTF-8 文本，每行一个停止词。例如
the
of
for
将文件上传到 Milvus 集群配置使用的对象存储，然后注册：
from pymilvus import MilvusClient
client = MilvusClient(uri="http://localhost:19530")
# Register the uploaded file under a name you'll reference from analyzer configs.
client.add_file_resource(
name="en_stop_words",
path="file/stop_words.txt", # full S3 object key, including the rootPath prefix
)
通过stop_words_file 在过滤器中引用已注册的资源：
analyzer_params = {
"tokenizer": "standard",
"filter": [{
"type": "stop",
"stop_words_file": {
"type": "remote",
"resource_name": "en_stop_words",
"file_name": "stop_words.txt",
},
}],
}
stop_words_file 参数接受包含以下字段的对象：
字段 |
字段 |
|---|---|
|
资源类型。通过 |
|
文件在 |
|
注册资源的对象存储路径中的文件名部分（例如，如果资源是通过 |
示例
在将分析器配置应用到 Collections 模式之前，请使用run_analyzer 方法验证其行为。
分析器配置
analyzer_params = {
"tokenizer": "standard",
"filter":[{
"type": "stop", # Specifies the filter type as stop
"stop_words": ["of", "to", "_english_"], # Defines custom stop words and includes the English stop word list
}],
}
Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
Collections.singletonList(
new HashMap<String, Object>() {{
put("type", "stop");
put("stop_words", Arrays.asList("of", "to", "_english_"));
}}
)
);
// javascript
analyzerParams = map[string]any{"tokenizer": "standard",
"filter": []any{map[string]any{
"type": "stop",
"stop_words": []string{"of", "to", "_english_"},
}}}
# restful
验证使用run_analyzer
from pymilvus import (
MilvusClient,
)
client = MilvusClient(uri="http://localhost:19530")
# Sample text to analyze
sample_text = "The stop filter allows control over common stop words for text processing."
# Run the standard analyzer with the defined configuration
result = client.run_analyzer(sample_text, analyzer_params)
print("Standard analyzer output:", result)
import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.service.vector.request.RunAnalyzerReq;
import io.milvus.v2.service.vector.response.RunAnalyzerResp;
ConnectConfig config = ConnectConfig.builder()
.uri("http://localhost:19530")
.build();
MilvusClientV2 client = new MilvusClientV2(config);
List<String> texts = new ArrayList<>();
texts.add("The stop filter allows control over common stop words for text processing.");
RunAnalyzerResp resp = client.runAnalyzer(RunAnalyzerReq.builder()
.texts(texts)
.analyzerParams(analyzerParams)
.build());
List<RunAnalyzerResp.AnalyzerResult> results = resp.getResults();
// javascript
import (
"context"
"encoding/json"
"fmt"
"github.com/milvus-io/milvus/client/v2/milvusclient"
)
client, err := milvusclient.New(ctx, &milvusclient.ClientConfig{
Address: "localhost:19530",
APIKey: "root:Milvus",
})
if err != nil {
fmt.Println(err.Error())
// handle error
}
bs, _ := json.Marshal(analyzerParams)
texts := []string{"The stop filter allows control over common stop words for text processing."}
option := milvusclient.NewRunAnalyzerOption(texts).
WithAnalyzerParams(string(bs))
result, err := client.RunAnalyzer(ctx, option)
if err != nil {
fmt.Println(err.Error())
// handle error
}
# restful
预期输出
['The', 'stop', 'filter', 'allows', 'control', 'over', 'common', 'stop', 'words', 'text', 'processing']