Skip to main content
This is in development. Details may change before it is available.
The semantic cache answers a request from an earlier answer when the request is the same as, or close enough in meaning to, one Altrum has answered before. The answer comes back faster and costs no model tokens. It works for text, chat and Responses requests, streamed or not.

​
Switch it on

The cache is off by default. Switching it on keeps your organization’s answers in its own cache for the time you set. That is separate from keeping prompts and answers in your logs, which has its own switch.
  • For organization and personal keys: an owner or admin switches it on in the AI Gateway settings.
  • For an AI system’s keys: switch it on or off on the AI system’s AI Gateway tab, or leave it to follow the organization’s setting.
These choices belong to the organization and apply to every AI system under it:

​
Boundaries

  • Each AI system has its own cache, and your organization’s other keys share one. A personal key reads and writes only its own person’s answers. An answer is never served across those lines, and never to another organization.
  • A cached answer still passes your guardrails on answers. If the guardrail changed after the answer was stored, the new rules apply to it.
  • Entries are deleted when their time is up. Owners and admins can also Clear cache at any time, for the organization’s keys or for one AI system.
  • If the cache is not available, requests are answered as if it were off.

​
What it costs and how it shows

  • A request answered from the cache is marked cache_hit in the request log and has no model cost.
  • Measuring similarity uses your provider credentials. Its small cost is added to the request that needed it and counts against your budgets. A request that repeats an earlier one word for word needs no measuring and costs nothing.

​
Request headers

A request can only make the cache stricter than your settings.