The foundational limitation for Large Language Models (LLMs) currently resides within their context window, the finite sequence of tokens an LLM can process at any given moment. This context window dictates the maximum length of input text – comprising both prompt and retrieved information – that the model can consider to generate a coherent and relevant response. While transformer models have revolutionized natural language processing (NLP) with their attention mechanisms,
