tokenization

How a CJK Bigram Tokenizer Powers Search in Nine Languages

Space-based tokenizers fail on Korean, Chinese, and Japanese text. Here is a bigram approach that searches nine languages with one index.