Decompounder

The decompounder filter splits compound words into individual components based on a specified dictionary, making it easier to search for parts of compound terms. This filter is particularly useful for languages that frequently use compound words, such as German.

Configuration

The decompounder filter is a custom filter in Milvus. To use it, specify "type": "decompounder" in the filter configuration, along with a word_list parameter that provides the dictionary of word components to recognize.

analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "decompounder", # Specifies the filter type as decompounder
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"],
    }],
}

The decompounder filter accepts the following configurable parameters.

The decompounder filter operates on the terms generated by the tokenizer, so it must be used in combination with a tokenizer.

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Example output

Here’s an example of how the decompounder filter processes text:

Original text:

"dampfschifffahrt brotbackautomat"

Expected output (with word_list: ["dampf", "schiff", "fahrt", "brot", "backen", "automat"]):

["dampf", "schiff", "fahrt", "brotbackautomat"]