IEEE

Improved Feature Tokenizer Transformer

Abstract: This paper proposes an improved FT-Transformer model to improve the efficiency of intrusion detection in computer networks. Key modifications include: integrating the ColumnEmbeddingAdder ...
GitHub

AutoPhrase: Automated Phrase Mining from Massive Text Corpora

Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, Jiawei Han, "Automated Phrase Mining from Massive Text Corpora", accepted by IEEE Transactions on Knowledge and Data Engineering, Feb.
IEEE

Zero-Shot Tokenizer Transfer for Targeted Attacks on Retrieval-Augmented Generation

Abstract: Retrieval-Augmented Generation (RAG) is widely used to improve the reliability of Large Language Models (LLMs) by grounding responses in external documents. However, RAG systems that ingest ...
GitHub

Minimal Tokenizer implementation of BertJapanese(cl-tohoku/bert-base-japanese) in C#

Just add BertJapaneseTokenizer package from Nuget. Download unidic mecab dictionary unidic-mecab-2.1.2_bin.zip from https://clrd.ninjal.ac.jp/unidic_archive/cwj/2.1.2 ...