English

The english analyzer in Milvus is designed to process English text, applying language-specific rules for tokenization and filtering.

Definition

The english analyzer uses the following components:

• Tokenizer: Uses the standard tokenizer to split text into discrete word units.

• Filters: Includes multiple filters for comprehensive text processing:

  • lowercase: Converts all tokens to lowercase, enabling case-insensitive searches.

  • stemmer: Reduces words to their root form to support broader matching (e.g., “running” becomes “run”).

  • stop_words: Removes common English stop words to focus on key terms in text.

The functionality of the english analyzer is equivalent to the following custom analyzer configuration:

analyzer_params = {
    "tokenizer": "standard",
    "filter": [
        "lowercase",
        {
            "type": "stemmer",
            "language": "english"
        }，{
            "type": "stop",
            "stop_words": "_english_",
        }
    ]
}

Configuration

To apply the english analyzer to a field, simply set type to english in analyzer_params, and include optional parameters as needed.

analyzer_params = {
    "type": "english",
}

The english analyzer accepts the following optional parameters:

Example configuration with custom stop words:

analyzer_params = {
    "type": "english",
    "stop_words": ["a", "an", "the"]
}

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Example output

Here’s how the english analyzer processes text.

Original text:

"The Milvus vector database is built for scale!"

Expected output:

["milvus", "vector", "databas", "built", "scale"]