Abstract: Extending the context window of large language models (LLMs) is crucial for effectively processing long texts, yet existing models typically struggle due to their restricted input lengths. Existing methods for extending context windows mainly include training-based and training-free approaches. However, training-based methods require significant computational resources, while training-free methods typically achieve limited performance. In this paper, we propose a new method to effectively extend the context window of LLMs. This method consists of three key components: First, we conduct an experimental analysis of Rotary Position Embedding (RoPE), a primary positional encoding method used in LLMs, to identify the critical characteristics that influence the context window size and performance of LLMs. Second, based on the analysis results, we formulate a mixed integer black-box optimization model and utilize its optimal solution to determine the best adjustment strategy for RoPE. Third, we develop an alternating Bayesian mixed integer optimization algorithm to solve the model. Extensive experiments show that our method successfully extends the context window of LLMs and outperforms most existing training-based and training-free methods across various tasks.

Key words: LLM context window extension, Mixed integer black-box optimization model, Alternating Bayesian mixed integer optimization algorithm, Rotary positional embedding