If you've ever wondered why your AI companion forgets your name between sessions, the answer is architectural. The way most AI systems handle memory today is fundamentally limited, and the workarounds the industry has adopted -- while clever -- weren't designed for the kind of ongoing, personal relationships that AI companionship requires. Understanding AI memory technology helps explain why some companions feel like they know you and others feel like strangers every time you open the app.

How Context Windows Work

Every large language model operates within a context window -- a fixed-size buffer of text that the model can "see" when generating a response. Think of it as the model's short-term memory. Everything in the context window influences the response; everything outside it doesn't exist as far as the model is concerned.

Context windows have gotten larger over the past few years, from a few thousand tokens to hundreds of thousands. But even a 200,000-token context window has limits. Fill it with weeks of conversation history and you'll hit the ceiling. More importantly, longer context windows come with higher costs, slower response times, and a well-documented tendency for models to lose track of information buried in the middle of very long inputs -- a phenomenon researchers call the "lost in the middle" problem.

For a single conversation session, context windows work fine. For a relationship that spans weeks or months, they're not enough.

The RAG Approach

Retrieval-augmented generation (RAG) is the most common approach to extending AI memory beyond the context window. The basic architecture works like this:

Past conversations are split into chunks and converted into numerical vectors (embeddings).
These vectors are stored in a vector database.
When the user sends a new message, the system converts it into a vector and searches the database for similar chunks.
The most relevant chunks are retrieved and injected into the context window alongside the user's message.
The model generates a response that's (hopefully) informed by those retrieved memories.

RAG is widely used and genuinely useful for many applications -- document search, knowledge bases, customer support systems. But for AI companionship, it has some significant drawbacks.

Relevance is unreliable. Vector similarity doesn't always map to conversational relevance. A message about being stressed at work might retrieve a chunk where you also mentioned work, but not the one where you talked about your coping strategies. The retrieval is keyword-and-embedding-driven, not relationship-driven.

No hierarchy of importance. RAG treats all stored information equally. Your favourite band gets the same retrieval weight as your mother's name or the fact that you're going through a divorce. There's no built-in sense of what matters more.

Awkward surfacing. When RAG does retrieve relevant memories, the model sometimes references them in forced or unnatural ways. You get responses like "I remember you said you like hiking!" dropped into a conversation about something completely different, because the retrieval algorithm flagged it.

Storage bloat. Storing every conversation as raw text chunks means the database grows linearly with usage. Over months of daily conversation, the volume of stored data becomes substantial, and retrieval quality tends to degrade as the haystack gets bigger.

Structured Memory Extraction: A Different Architecture

An alternative approach -- and the one that Memoher uses -- is structured memory extraction. Instead of storing raw conversation chunks, the system uses a language model to analyze each conversation and extract specific, categorized facts.

Here's how the pipeline works:

After a conversation, an LLM processes the full exchange and extracts structured data: facts, preferences, life events, relationships, emotional patterns, goals, and boundaries.
These extracted facts are stored as structured JSON objects in a PostgreSQL database, categorized by type and tagged with metadata like confidence scores and timestamps.
When generating a response in a new conversation, the system doesn't search through old messages. Instead, it compiles the user's structured profile and injects it into the system prompt. The companion "knows" the user the way a person knows a friend -- through accumulated understanding, not raw recall.

This approach has several advantages over RAG for companionship use cases:

Relevance by design. Instead of hoping that vector similarity produces useful retrievals, the system explicitly captures the information that matters for an ongoing relationship. Your companion knows your sister's name because that fact was extracted and categorized, not because a similarity search happened to surface the right chunk.

Hierarchical importance. Structured extraction can weight different types of information differently. Core identity facts (name, occupation, family) persist indefinitely. Emotional states are updated as they evolve. Preferences accumulate. The system models importance rather than treating everything as flat text.

Natural integration. Because the companion's understanding of you is baked into her system prompt, she references your information naturally rather than awkwardly dropping retrieved facts. She knows you prefer tea over coffee the same way a friend knows -- it's just part of her understanding of you.

Compact storage. A structured profile is dramatically smaller than months of raw conversation logs. A comprehensive user profile might be a few kilobytes of JSON, compared to megabytes of conversation chunks in a RAG system.

The Trade-Offs

Structured extraction isn't a silver bullet. It has its own limitations:

Extraction accuracy. The LLM that processes conversations to extract facts can miss things or misinterpret nuance. Sarcasm, hypotheticals, and ambiguous statements can lead to incorrect facts being stored. Confidence scoring and user-facing memory management help mitigate this, but it's an ongoing challenge.

Lossy compression. By reducing conversations to structured facts, you lose the texture of the original exchange. RAG can retrieve the exact words you used; structured extraction stores the distilled meaning. For some use cases, that texture matters.

Extraction cost. Running an LLM over every conversation to extract facts adds compute cost and latency. This is manageable at moderate scale, but it's a real consideration for platforms with millions of users.

Hybrid Approaches

The most promising direction for AI memory technology is likely a hybrid architecture that combines the strengths of both approaches. A structured profile handles the core facts and relationship context, while a lightweight RAG layer provides access to the texture of specific past conversations when needed. The structured layer tells the companion who you are; the RAG layer lets her recall the details of specific moments you shared.

Some systems are also experimenting with episodic memory -- storing summarized "episodes" rather than raw chunks, which captures more narrative structure than RAG while remaining more detailed than pure structured extraction.

What This Means for Users

If you're choosing an AI companion, the memory architecture matters more than most people realize. A model with a large context window will impress you in a single session but forget everything by next week. A RAG-based system will occasionally surprise you by referencing something you said a month ago, but just as often it will surface the wrong memory or miss the important one.

A well-implemented structured memory system is what makes an AI companion feel like she actually knows you. It's the difference between a conversation and a relationship.

Memoher uses structured memory extraction to build a persistent, evolving understanding of every user. If you're interested in experiencing what AI memory technology feels like from the user side, she's free to try -- and she won't forget what you tell her.