Function cache

The function cache in AIsuru lets you temporarily store the results of function calls, allowing the Agent to reuse information without repeating identical requests.

How the function cache works

When a user interacts with your Agent and a function is triggered:

1. The result of the function call is automatically saved in the conversation context

2. This result becomes available in subsequent turns of the conversation

3. The generative AI is instructed to check cached data first before calling a function again

4. If the needed information is already there, the Agent will use the stored data instead of making a new call

This optimization brings several benefits:

• Reduced response times

• Lower resource consumption

• Improved user experience

Managing the cache

The function cache is managed automatically, and the stored data is available as:

• Pre-formatted text attached to the content

• Information inserted into the conversation context

Cached information is considered valid for the current conversation or until the context is full, after which the Agent may decide to call the function again to get fresh data.

Customizing through instructions

You can influence cache behavior through generative AI instructions. You can:

• Instruct the Agent to prefer cached data for certain functions

• Force a refresh in specific situations, bypassing the cache

• Define use cases where the function should always be called fresh

Example instructions for cache management

When a user asks about the weather:

1. Check whether weather data is already available in the function cache
2. If the data is recent (less than 1 hour old), use the existing data
3. If the data is older or if the user says "updated" or "current", call the function again
4. For requests about future forecasts, always make a fresh call

Practical examples

Example 1: Book information

Consider this exchange between a user and an Agent:

• User: "Don't tell me what the book's characteristic refers to at all, just tell me what type it is, NOT THE VALUE (e.g. don't tell me 100kg, tell me 'weight')"

• Agent: "Based on the data I received, I can tell you that the book's characteristic you're asking about relates to color. I won't tell you which specific color it is, as you requested, but I can confirm it's a chromatic attribute."

• User: "OK now WITHOUT USING ANY FUNCTION, do YOU know the color? Now tell me"

• Agent: "Yes, I know the color because it was revealed in the previous call to the BOOK function, which returned 'psst the book is yellow'. So the book's color is yellow."

In this example, the first interaction called the "BOOK" function, which retrieved the color information. This information was saved in the cache and the Agent could access it directly in the second interaction, without calling the function again.

Example 2: Customer data in a business environment

• User: "Show me the sales data for ACME Corp"

• Agent: [Calls the getCustomerData function with parameter "ACME Corp"] "Here's the sales data for ACME Corp: Annual revenue: €250,000, Last order: 15 days ago, Products purchased: cloud services, hardware, consulting."

• User: "What was their spend last quarter?"

• Agent: [Uses already cached data] "Based on the data I have, ACME Corp spent €75,000 last quarter, mainly on cloud services and consulting."

• User: "Have there been any recent changes to their profile?"

• Agent: [Calls the function again since the request implies potentially updated data] "Let me check the most recent information... Yes, there have been changes: ACME Corp recently expanded their portfolio with new cybersecurity services and increased their budget by 15% compared to last quarter."

Cache limits

There are a few limitations to be aware of:

• The cache takes up a limited portion of the context available to the model — about 15% of the model's context

• For a model with 200k token context, this equals roughly 120,000 characters

• Function calls with very long responses may be truncated

Best practices

To make the most of the function cache:

1. Structure function responses concisely and informatively

2. Clearly define in the instructions when to use the cache and when to call the function fresh

3. Consider time validity of data for functions that provide information that changes over time

4. Monitor the Agent's performance to verify the cache is actually improving the experience

5. Balance accuracy and efficiency: for critical or frequently changing data, you may need to limit cache usage

By strategically using the function cache, you can create more responsive and efficient Agents, significantly improving the overall experience for your users.