Memory Agent: A Shared Knowledge Fabric for AI Agents

1. Executive summary

Purpose, intended audience, and principal conclusions.

AI agents are becoming capable participants in business and engineering workflows, but most still operate with fragmented context. One agent discovers a useful fact, another repeats the same investigation, and a third produces an answer without knowing where its evidence originated. Memory Agent addresses that fragmentation through a shared, governed memory layer.

The system consolidates knowledge produced across a user’s agent sessions and makes approved knowledge reusable across a team. It combines authenticated ingestion, independent vector and memory processing, multimodal understanding, knowledge relationships, source verification, and operational auditing behind a standard agent interface.

The central conclusion of this paper is that useful agent memory requires more than semantic similarity. A production memory layer must preserve identity, scope, processing state, ownership, contributor attribution, and verifiable source evidence throughout the complete lifecycle.

2. Problem statement and design objectives

The operational problems addressed by Memory Agent and the requirements derived from them.

The project began with two related problems. The first exists inside a single user’s workflow. A developer may use several agents, projects, and sessions during one week, but the knowledge produced in those conversations is scattered across temporary contexts.

The second problem appears at team scale. Useful knowledge may belong to one contributor while being relevant to everyone. Sharing it must not erase ownership, expose private material, or reduce the result to an unattributed answer.

Design objectives

• Consolidate durable knowledge across multiple agents, projects, and sessions without sharing transient conversation state.
• Keep private retrieval user-scoped while enabling explicitly governed project and team knowledge.
• Preserve ownership, contributor attribution, source filenames, and processing history in every shared result.
• Support text, documents, and images through a consistent multimodal ingestion contract.
• Separate write-capable memory extraction from read-only retrieval and reasoning.
• Remain independent of a specific vector engine, embedding model, agent client, or model-serving provider.

3. Reference architecture

Logical components, trust boundaries, and separation of responsibilities.

Clients connect through the Model Context Protocol. The public boundary authenticates every request and resolves the user and session before any internal worker is called. This means coding, research, and workflow agents can use the same small set of tools without learning the storage topology behind them.

The project deliberately separates the Add Agent from the Search Agent. Adding knowledge is a write operation: it extracts durable facts, produces summaries, records provenance, and builds graph relationships. Searching is read-only: it ranks existing evidence and explains why each result matters.

4. Knowledge ingestion and processing lifecycle

Content ingestion, deduplication, independent processing, and completion semantics.

Text, Markdown, PDF, and image files enter through a user-authenticated upload flow. The service calculates a content identifier, stores the bytes once, and creates a separate processing record for each user. Global byte-level deduplication therefore does not collapse user ownership or processing state.

Two workers then operate independently:

VectorDB extracts searchable chunks and embeddings. Agent Add reads or visually inspects the source, writes durable memory records, and adds user-scoped and shareable graph knowledge. A relational status database is the shared processing authority, so either worker can retry or repair its own output without corrupting the other path.

Why multimodal processing matters

A PDF is more than extracted text, and an image is more than a filename. Memory Agent can supply PDF pages and uploaded images to a multimodal model so layout, diagrams, labels, and visual relationships can become retrievable memory. The original file remains available as the final verification source.

5. Retrieval and evidence synthesis

A staged retrieval model that expands scope only when evidence justifies it.

A similarity score alone is not enough for high-trust memory. Agent Search uses a retrieval ladder that begins with the requesting user’s durable memory and expands only when the evidence justifies it.

The final result separates direct vector matches from Agent Memory conclusions. Every Agent Memory chunk includes content, a source filename, a contributor, and a comment explaining why that evidence relates to the query.

A useful memory answer should say not only what the system remembers, but also whose knowledge it is, where it came from, and why it answers this question.

6. Identity, governance, and auditability

Controls required to make agent memory safe and accountable in a multi-user environment.

Memory becomes valuable only when users can trust its boundaries. Memory Agent supports browser and MCP authentication, separate client sessions, and an audit dashboard showing sessions, files, processing outcomes, request durations, and tool activity.

This is particularly important for teams. A shared result should retain contributor provenance. A file should remain associated with its owner even when its bytes were deduplicated. An administrator should be able to understand when an upload failed, retry one processing path, revoke a session, and inspect which tools were used.

7. Model infrastructure and deployment flexibility

Workload-specific models, deployment independence, and operational control.

The Agent services use a standard model-serving API and can assign different models to different jobs. A larger multimodal model can perform careful memory extraction, while a smaller model handles latency-sensitive searches. Model weights are cached independently from application data, and GPU-serving infrastructure can evolve without changing the client contract.

This separation creates useful choices: local models for privacy and cost control, remote model endpoints for elasticity, and workload-specific models for extraction versus retrieval.

8. Current limitations and implementation roadmap

Known boundaries of the current implementation and the program required for production maturity.

The current implementation demonstrates the end-to-end product experience, but the target architecture requires additional controls before broad enterprise deployment. The most significant limitation is that shared knowledge must evolve from a system-wide global scope into explicit organization, team, and project scopes. Sharing decisions should be policy-controlled and reviewable rather than delegated solely to model judgment.

Processing dispatch must also become durable so accepted work survives service restarts. Credentials require external secret management, local passwords require strong one-way hashing, dependencies must be pinned in immutable images, and production storage must support backup, restore, and multi-node recovery.

9. Demonstration workflows

Representative terminal sessions showing single-agent knowledge consolidation and cross-agent knowledge sharing.

The following recorded workflows illustrate how an agent identifies its session, discovers the available tools, adds or uploads source material, and compares direct VectorDB retrieval with deeper Agent Memory search. The second workflow demonstrates how knowledge contributed through one agent can be retrieved through another while retaining source and contributor evidence.

Loading demo…