Stop

The stop filter removes specified stop words from tokenized text, helping to eliminate common, less meaningful words. You can configure the list of stop words using the stop_words parameter.

Configuration

The length filter is a custom filter in Milvus. To use it, specify "type": "stop" in the filter configuration, along with a stop_words parameter that provides a list of stop words.

analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "stop", # Specifies the filter type as stop
        "stop_words": ["of", "to", "_english_"], # Defines custom stop words and includes the English stop word list
    }],
}

The stop filter accepts the following configurable parameters.

The stop filter operates on the terms generated by the tokenizer, so it must be used in combination with a tokenizer.

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Example output

Here’s an example of how the stop filter processes text:

Original text:

"The stop filter allows control over common stop words for text processing."

Expected output (with stop_words: ["the", "over", "_english_"]):

["The", "stop", "filter", "allows", "control", "common", "stop", "words", "text", "processing"]