Stop
The stop filter removes specified stop words from tokenized text, helping to eliminate common, less meaningful words. You can configure the list of stop words using the stop_words parameter.
Configuration
The stop filter accepts its stop-words list either inline via the stop_words parameter or from a registered file resource via the stop_words_file parameter.
Inline stop-words list
To use the stop filter with an inline list, specify "type": "stop" in the filter configuration, along with a stop_words parameter that provides the list of stop words.
analyzer_params = {
"tokenizer": "standard",
"filter":[{
"type": "stop", # Specifies the filter type as stop
"stop_words": ["of", "to", "_english_"], # Defines custom stop words and includes the English stop word list
}],
}
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
Collections.singletonList(
new HashMap<String, Object>() {{
put("type", "stop");
put("stop_words", Arrays.asList("of", "to", "_english_"));
}}
)
);
analyzerParams := map[string]any{"tokenizer": "standard",
"filter": []any{map[string]any{
"type": "stop",
"stop_words": []string{"of", "to", "_english_"},
}}}
use milvus::v2::prelude::*;
use serde_json::json;
let analyzer_params = json!({
"tokenizer": "standard",
"filter": [{
"type": "stop",
"stop_words": ["of", "to", "_english_"]
}]
});
nlohmann::json analyzer_params = {
{"tokenizer", "standard"},
{"filter", {
{{"type", "stop"}, {"stop_words", {"of", "to", "_english_"}}}
}}
};
const analyzer_params = {
tokenizer: "standard",
filter: [{
type: "stop", // Specifies the filter type as stop
stop_words: ["of", "to", "_english_"], // Custom stop words + built-in English list
}],
};
using System.Collections.Generic;
Dictionary<string, object> analyzerParams = new()
{
["tokenizer"] = "standard",
["filter"] = new List<object>
{
new Dictionary<string, object>
{
["type"] = "stop", // Specifies the filter type as stop
["stop_words"] = new List<object> { "of", "to", "_english_" } // Defines custom stop words and includes the English stop word list
}
}
};
# restful
analyzerParams='{
"tokenizer": "standard",
"filter": [
{
"type": "stop",
"stop_words": [
"of",
"to",
"_english_"
]
}
]
}'
The stop filter accepts the following configurable parameters.
Parameter |
Description |
|---|---|
|
A list of words to be removed from tokenization. By default, the filter uses the built‑in
|
The stop filter operates on the terms generated by the tokenizer, so it must be used in combination with a tokenizer. For a list of tokenizers available in Milvus, refer to Standard Tokenizer and its sibling pages.
After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.
Load stop words from a file resourceCompatible with Milvus 3.0.x
For large custom stop-words lists — language-specific lists, domain vocabularies, or lists you want to share across many collections — store the words in a file and register the file as a remote file resource, then reference it from the filter via the stop_words_file parameter. You can use stop_words_file on its own or alongside inline stop_words; when both are set, the filter merges the two sources into a single stop-words list.
The file is plain UTF‑8 text with one stop word per line. For example:
the
of
for
Upload the file to the object store that your Milvus cluster is configured to use, then register it:
from pymilvus import MilvusClient
client = MilvusClient(uri="http://localhost:19530")
# Register the uploaded file under a name you'll reference from analyzer configs.
client.add_file_resource(
name="en_stop_words",
path="file/stop_words.txt", # full S3 object key, including the rootPath prefix
)
Reference the registered resource in the filter via stop_words_file:
analyzer_params = {
"tokenizer": "standard",
"filter": [{
"type": "stop",
"stop_words_file": {
"type": "remote",
"resource_name": "en_stop_words",
"file_name": "stop_words.txt",
},
}],
}
The stop_words_file parameter accepts an object with the following fields:
Field |
Description |
|---|---|
|
The resource type. Use |
|
The name used when the file was registered with |
|
The filename portion of the registered resource's object-store path (for example, |
Examples
Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.
Analyzer configuration
analyzer_params = {
"tokenizer": "standard",
"filter":[{
"type": "stop", # Specifies the filter type as stop
"stop_words": ["of", "to", "_english_"], # Defines custom stop words and includes the English stop word list
}],
}
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
Collections.singletonList(
new HashMap<String, Object>() {{
put("type", "stop");
put("stop_words", Arrays.asList("of", "to", "_english_"));
}}
)
);
analyzerParams := map[string]any{"tokenizer": "standard",
"filter": []any{map[string]any{
"type": "stop",
"stop_words": []string{"of", "to", "_english_"},
}}}
use milvus::v2::prelude::*;
use serde_json::json;
let analyzer_params = json!({
"tokenizer": "standard",
"filter": [{
"type": "stop",
"stop_words": ["of", "to", "_english_"]
}]
});
nlohmann::json analyzer_params = {
{"tokenizer", "standard"},
{"filter", {
{{"type", "stop"}, {"stop_words", {"of", "to", "_english_"}}}
}}
};
const analyzer_params = {
tokenizer: "standard",
filter: [{
type: "stop", // Specifies the filter type as stop
stop_words: ["of", "to", "_english_"], // Custom stop words + built-in English list
}],
};
using System.Collections.Generic;
Dictionary<string, object> analyzerParams = new()
{
["tokenizer"] = "standard",
["filter"] = new List<object>
{
new Dictionary<string, object>
{
["type"] = "stop", // Specifies the filter type as stop
["stop_words"] = new List<object> { "of", "to", "_english_" } // Defines custom stop words and includes the English stop word list
}
}
};
# restful
analyzerParams='{
"tokenizer": "standard",
"filter": [
{
"type": "stop",
"stop_words": [
"of",
"to",
"_english_"
]
}
]
}'
Verification using run_analyzer
from pymilvus import (
MilvusClient,
)
client = MilvusClient(uri="http://localhost:19530", token="root:Milvus")
# Sample text to analyze
sample_text = "The stop filter allows control over common stop words for text processing."
# Run the analyzer with the defined configuration
result = client.run_analyzer(sample_text, analyzer_params)
print("Analyzer output:", result)
import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.service.vector.request.RunAnalyzerReq;
import io.milvus.v2.service.vector.response.RunAnalyzerResp;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
ConnectConfig config = ConnectConfig.builder()
.uri("http://localhost:19530")
.token("root:Milvus")
.build();
MilvusClientV2 client = new MilvusClientV2(config);
List<String> texts = new ArrayList<>();
texts.add("The stop filter allows control over common stop words for text processing.");
RunAnalyzerResp resp = client.runAnalyzer(RunAnalyzerReq.builder()
.texts(texts)
.analyzerParams(analyzerParams)
.build());
List<RunAnalyzerResp.AnalyzerResult> results = resp.getResults();
import (
"context"
"fmt"
"github.com/milvus-io/milvus/client/v3/milvusclient"
)
ctx := context.Background()
client, err := milvusclient.New(ctx, &milvusclient.ClientConfig{
Address: "http://localhost:19530",
APIKey: "root:Milvus",
})
if err != nil {
fmt.Println(err.Error())
// handle error
}
texts := []string{"The stop filter allows control over common stop words for text processing."}
option := milvusclient.NewRunAnalyzerOption(texts...).
WithAnalyzerParams(analyzerParams)
result, err := client.RunAnalyzer(ctx, option)
if err != nil {
fmt.Println(err.Error())
// handle error
}
fmt.Println(result)
use milvus::v2::prelude::*;
use serde_json::json;
let config = ConnectConfig::new().uri("http://localhost:19530").token("root:Milvus");
let client = ClientV2::new(&config).await?;
let analyzer_params = json!({
"tokenizer": "standard",
"filter": [{
"type": "stop",
"stop_words": ["of", "to", "_english_"]
}]
});
let result = client
.run_analyzer(
RunAnalyzerRequest::builder()
.texts(vec!["The stop filter allows control over common stop words for text processing."])
.analyzer_params(analyzer_params)
.build()?,
)
.await?;
#include "milvus/MilvusClientV2.h"
#include <iostream>
#include <string>
auto client = milvus::MilvusClientV2::Create();
milvus::ConnectParam connect_param{"http://localhost:19530", "root:Milvus"};
auto status = client->Connect(connect_param);
if (!status.IsOk()) {
std::cout << status.Message() << std::endl;
}
std::string text = "The stop filter allows control over common stop words for text processing.";
auto request = milvus::RunAnalyzerRequest()
.AddText(text)
.WithAnalyzerParams(analyzer_params);
milvus::RunAnalyzerResponse response;
status = client->RunAnalyzer(request, response);
if (!status.IsOk()) {
std::cout << status.Message() << std::endl;
}
import { MilvusClient } from "@zilliz/milvus2-sdk-node";
const client = new MilvusClient({
address: "http://localhost:19530",
token: "root:Milvus",
});
const sample_text = "The stop filter allows control over common stop words for text processing.";
// Run the analyzer with the defined configuration
const result = await client.runAnalyzer({
text: sample_text,
analyzer_params,
});
console.log("Analyzer output:", result);
using Milvus.Client.V2;
using Milvus.Client.V2.Requests.Utility;
using Milvus.Client.V2.Responses.Utility;
using Milvus.Client.V2.Types;
MilvusClientV2 client = new(new ConnectConfig
{
Uri = "http://localhost:19530",
ApiKey = "root:Milvus"
});
// Sample text to analyze
string sampleText = "The stop filter allows control over common stop words for text processing.";
// Run the analyzer with the defined configuration
RunAnalyzerResp result = await client.RunAnalyzerAsync(new RunAnalyzerReq
{
AnalyzerParams = analyzerParams,
Texts = [sampleText]
});
foreach (AnalyzerToken t in result.Results.SelectMany(r => r.Tokens))
{
Console.WriteLine(t.Token);
}
# restful
curl --request POST \
--url "http://localhost:19530/v2/vectordb/common/run_analyzer" \
--header "Authorization: Bearer root:Milvus" \
--header "Content-Type: application/json" \
-d '{
"text": ["The stop filter allows control over common stop words for text processing."],
"analyzerParams": "{\"tokenizer\": \"standard\", \"filter\": [{\"type\": \"stop\", \"stop_words\": [\"of\", \"to\", \"_english_\"]}]}"
}'
Expected output
['The', 'stop', 'filter', 'allows', 'control', 'over', 'common', 'stop', 'words', 'text', 'processing']