← all papers · overview

IGOT: Information Gain Optimized Tokenizer On Domain Adaptive Pretraining

Abstract

Pretrained Large Language Models (LLM) such as ChatGPT, Claude, etc. have demonstrated strong capabilities in various fields of natural language generation. However, there are still many problems when using LLM in specialized domain-specific fields. When using generative AI to process downstream tasks, a common approach is to add new knowledge (e.g., private domain knowledge, cutting-edge informat

Related papers

Ranked by semantic similarity — how closely each paper's abstract matches this one (100% = near-identical topic).