Short-Term / Conversation Memory

Core concepts

1. Context window basics: Everything the model "remembers" in a single turn is whatever's in the context window — nothing more, nothing less.
2. Conversation history management: Naively appending every message eventually exceeds the context window — most frameworks provide a message-history abstraction that trims or summarizes.
3. Context compaction: Summarizing or selectively dropping old turns to stay under budget without losing information the agent still needs — a genuinely hard problem, covered in depth on our own Jev breakdown, where a decision model is used to keep or drop old tool calls without ever rewriting them.
4. Prompt caching: Most providers now support caching the static part of your prompt (system instructions, tool schemas) so repeated calls don't re-process the same tokens every time — see OpenAI Prompt Caching and Anthropic prompt caching.

What Is It?

Short-term memory is everything currently sent to the model: system prompt, tools, and conversation history.

Techniques include sliding windows, summarization, and structured scratchpads.

Prompt caching reduces cost/latency for repeated prefixes.

Why Should I Care?

Tool loops multiply messages quickly — unmanaged history explodes bills.

Lost context makes agents forget constraints mid-task.

Real World Analogy

The context window is a whiteboard with limited space — you erase oldest doodles (summarize or drop) when the board fills up.

Visual Mental Model

messages -> trim/summarize -> fit window
  prompt cache (optional)

Beginner Explanation

Manage history length, summarize old turns, and use prompt caching where providers allow. Every token in history is rent you pay each step.

Smallest Possible Example

def trim(messages, max_turns=6):
    return messages[-max_turns:]

Common Mistakes

❌ Never summarizing long support threads.

Eventually you truncate critical instructions.

❌ Caching mutable tool outputs.

Stale cache sends wrong facts to the model.

❌ Putting secrets in system prompts that get logged.

History replication exposes them to analytics tools.

Real Project Example

OpenAI prompt caching benefits agents with stable system prompts.

Coding agents like Cursor trim/file-select context aggressively each turn.

Build This Mini Project

Implement sliding-window trim on a list of message dicts; log token estimate before/after.

Interview Question

How do you fit long chats into context limits?

Trim oldest turns, summarize middle history into a rolling brief, keep system and recent tool results, and optionally store overflow in long-term retrieval instead of inline text.

Key Takeaways

• Context window is expensive RAM.
• Summarize and trim deliberately.
• Use caching for stable prefixes.
• Protect secrets from log pipelines.

Resources

• Anthropic: Context Editing & Memory
• OpenAI: Prompt Caching

YouTube learning

Core concepts from video walkthroughs:

• Context windows & embeddings (semantic “memory”)
• Managing context with retrieval (RAG)
• Langfuse — tracing long agent runs & prompts

The 60-second story

A waiter can only hold so many specials in their head while they are at your table. That head is the context window. If it is not in their head for this visit, they do not “remember” it, no matter how charming the training was. Short-term memory is not a diary in a drawer. It is whatever you managed to keep on the tray for this turn.

If you tape every previous sentence to the tray, the tray fills. The salad from lunch on Tuesday crowds out the allergy the guest just stated. History management is deciding what stays on the tray. Some frameworks trim from the front. Better ones summarize. The hard version is compaction: shrink old turns without losing the one fact that still matters, like a coupon code or a tool result you must not repeat.

There is a nasty version of “summarize” that rewrites old tool calls into a story and then the model believes the story instead of the tool. A careful approach keeps the old slips and decides which slips are still on the tray. Dropping is safer than fan fiction.

Prompt caching is the printer that does not re-ink the permanent menu every time a guest orders. System instructions and tool schemas are the permanent menu. Providers can cache that prefix so you stop paying full price to reread your own poster. The guest’s latest sentence is the part that changes.

Map the table.

• The context window is the waiter’s head for this visit.
• Naive history is taping every sentence to the tray until it tips.
• Compaction is shrinking the past on purpose.
• Rewriting old tool results is fan fiction. Avoid it.
• Prompt caching is not re-inking the permanent menu.

tray = system + tools + recent turns + the one old fact you still need

That is the whole architecture in one line. Everything else is how you choose the last item.

Short-term memory exists because models are stateless between calls unless you bring the past back. The window exists because “bring all of it” became too expensive and too distracting. Caching exists because teams noticed they were paying to reread a menu that had not changed.

Mental model: a tray, not a diary. If it is not on the tray this turn, the model does not know it. Your job is editing the tray, not hoping for a bigger head.

You have the page if you can explain why appending forever fails, and why a summary that rewrites tool output can lie.