Selective Memory in AI Agents

Research question

Does selective memory improve an AI agent’s ability to answer questions about changing, outdated, and canceled project information?

Study setup

We tested three memory ways for answering questions about a changing software project: large-context prompting, traditional TF-IDF RAG, and selective memory. The benchmark used 90 events that occurred and 30 questions involving changing requirements, task ownership, blockers, canceled work, outdated information, and source-event identification.

The benchmark used the frontier AI model GPT 5.6 Sol developed by OpenAI with medium reasoning. The large context group received the full project history at once, and the RAG system used TF-IDF to select the top 5 most relevant events for each question. The selective memory system used a table with columns for current, updated, outdated, canceled, usable, and ignorable information.

Results

Selective memory was the most accurate of the 3, answering 27 out of 30 questions correctly (90.0%). On the other hand, the large context group answered 26 out of 30 questions correctly for an accuracy of 86.7%. Lastly, the RAG system answered 24 out of 30 questions correctly, for an accuracy of 80%.

Audited comparison of the three approaches

Selective memory was the overall accuracy winner, while RAG was the fastest approach. Large context had 4 wrong answers and selective memory had 3. In the canceled category, selective memory made zero mistakes, while both large context and RAG made one mistake.

Selective memory performed best on questions involving ownership changes, blocker transitions, and canceled features because it was provided with the specific table containing these items. It was the most accurate approach, with accurate results for Q16, Q17, and Q25; the main error was in Q23.

RAG pulled at least one supporting event for 28 out of 30 questions; it fully pulled the correct supporting events for 23 questions and a partial set for the remaining 5, and did not pull any events for 2 questions.

In addition, both selective memory and large context were slower than RAG, with RAG taking 108.809 seconds, large context 150.057 seconds, and selective memory 217.326 seconds.

Discussion

The benchmark results show that selective memory in AI agents can help manage an evolving project context. Even though it is slower, its accuracy is thanks to the explicitly separated table. This may reduce hallucinations caused by stale data and canceled features in AI agents’ memory.

Selective memory was not perfect. Its Q23 error shows that remembering the correct topic is not the same as identifying the exact source events. Its partially correct answers also show that a memory table can still contain too much related information or confuse details involving multiple owners. Selective memory is useful for reasoning, but it does not guarantee a perfect answer every time.

RAG had important strengths compared to the other 2 ways; it was the fastest method, and it retrieved supporting events for the majority of questions. However, the retrieved information was not always correct, and it did not always help the AI agent to answer the question correctly. The pulled context often lacked an important event needed to answer the question correctly.

Large-context prompting provided access to the full project context, allowing the model to answer questions correctly. However, the model had to work harder to find the needed information among all the given data, making it more vulnerable to stale context poisoning and causing the run to take longer.

As a result, selective memory produced the most accurate answers, even though it took the model longer to answer the questions. However, this experiment does not prove that selective memory is always better than large-context prompting or RAG.

Limitations

This experiment used a dataset comprising 90 project events and 30 questions, so the results may not generalize to other companies or other types of long-term AI agent tasks.

The RAG system used a simple TF-IDF retriever, returning five events. More advanced embedding-based or hybrid retrieval systems might perform differently. Selective memory was implemented as an explicit structured table, so the comparison does not represent every possible memory architecture.

Future directions

In the future, selective memory should be evaluated on larger datasets and more advanced implementations, with more questions and trials. The benchmark could also be tested on real-life data to get more accurate results.

Also, multiple models with varying levels of reasoning effort could be used to assess the accuracy of selective memory across models. Future experiments should separately measure answer correctness, retrieval coverage, exact source-event accuracy, outdated-information errors, latency, token usage, and actual monetary cost.

The benchmark could also compare selective memory with a memory architecture that combines structured memory and semantic retrieval. Additional questions should test whether an agent can correctly update its memory when requirements change, forget canceled work, preserve exact source provenance, and avoid including irrelevant historical information.

• What should we test next?
• How could we use more data or more models?
• How could we improve retrieval, memory, speed, and cost measurement?

Sources and citation notes

The results in this draft come from the project repository, especially:

• results/week7_summary.md
• results/week7_final_scoring.csv
• results/week7_manual_review.csv
• visual_benchmark/benchmark.svg