elasticsearch/docs/reference/elasticsearch-plugins/analysis-kuromoji-analyzer.md

mapped_pages
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji-analyzer.html

kuromoji analyzer [analysis-kuromoji-analyzer]

The kuromoji analyzer uses the following analysis chain:

• CJKWidthCharFilter from Lucene
• kuromoji_tokenizer
• kuromoji_baseform token filter
• kuromoji_part_of_speech token filter
• ja_stop token filter
• kuromoji_stemmer token filter
• lowercase token filter

It supports the mode and user_dictionary settings from kuromoji_tokenizer.

Normalize full-width characters [kuromoji-analyzer-normalize-full-width-characters]

The kuromoji_tokenizer tokenizer uses characters from the MeCab-IPADIC dictionary to split text into tokens. The dictionary includes some full-width characters, such as ｏ and ｆ. If a text contains full-width characters, the tokenizer can produce unexpected tokens.

For example, the kuromoji_tokenizer tokenizer converts the text Ｃｕｌｔｕｒｅ　ｏｆ　Ｊａｐａｎ to the tokens [ culture, o, f, japan ] instead of [ culture, of, japan ].

To avoid this, add the icu_normalizer character filter to a custom analyzer based on the kuromoji analyzer. The icu_normalizer character filter converts full-width characters to their normal equivalents.

First, duplicate the kuromoji analyzer to create the basis for a custom analyzer. Then add the icu_normalizer character filter to the custom analyzer. For example:

PUT index-00001
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "kuromoji_normalize": {                 <1>
            "char_filter": [
              "icu_normalizer"                    <2>
            ],
            "tokenizer": "kuromoji_tokenizer",
            "filter": [
              "kuromoji_baseform",
              "kuromoji_part_of_speech",
              "cjk_width",
              "ja_stop",
              "kuromoji_stemmer",
              "lowercase"
            ]
          }
        }
      }
    }
  }
}

1. Creates a new custom analyzer, kuromoji_normalize, based on the kuromoji analyzer.
2. Adds the icu_normalizer character filter to the analyzer.