As I explained in Episode 5, LLMs are limited by their context window, or how much data they can handle. Asking the LLM to process data from multiple data sources usually fails because the dataset is too big. In Episode 5, I proposed an easy solution: constrain the user's questions to data that fits within the context window.

While this is fine for a first release, you need an approach for processing datasets bigger than the context window can handle. This is possible with a Scratchpad, an area where the LLM Agent can store temporary information while working through complex problems. Watch Episode 12 to see how Scratchpads work. 

Here is a summary.

A user asks a question that cannot be answered without processing more data than the LLM can handle. An LLM Agent starts by creating a plan, and the plan requires the Agent to execute a series of steps. 

The Agent has access to a set of tools and can select the right tool for each step. Tools are common Python data functions such as selecting data from tables or finding the largest value in a set of numbers.

The Agent uses tools to create temporary sets of information in memory, called Variables. Variables are stored in the Scratchpad. 

Why is it called the Scratchpad technique? Because the Agent generates the Variables dynamically for every question. This conceptually simple design is incredibly powerful because the LLM can reason through problems and develop the right Variables for the right problems. 

I skip over many engineering details in this episode. Justin and I will dive into more detail next week.

Enjoy!

p.s. - Be sure to watch Episode 5 for more background on LLM context windows.