Explains the two distinct memory layers in AI agents—working memory files and a searchable knowledge vault—and how to use each correctly.
Adapted from @MichaelGannotti# Your Agent Has Two Brains. Stop Treating Them as One MEMORY.md is not a second brain. Hermes and OpenClaw both “remember.” That word hides two different systems. One is the agent’s working memory: tiny curated files like MEMORY.md and USER.md that ride into the prompt. The other is a memory vault or second brain: an Obsidian vault, LLM wiki, or markdown knowledge base the agent can search and maintain. It should not be dumped into every turn. If you collapse those two layers, the agent either forgets who you are, or it drags a novel into every reply and compresses the important stuff away. The two layers Think of it like a person, not a chatbot. Agent memory is always-on working memory. Good examples: MEMORY.md, USER.md, and skills. Bad use: dumping a wiki into the system prompt. A memory vault is on-demand knowledge. Good examples: an Obsidian vault, an LLM wiki, or dated daily notes. Bad use: expecting the whole vault to load into every reply. They solve different problems. 1. Agent memory (MEMORY.md) This is the sticky note on the monitor. In Hermes it is usually two small files: • MEMORY.md for durable facts, standing decisions, and project pointers • USER.md for you: preferences, communication style, and constraints Those files are deliberately tiny. Community writeups put Hermes around 2,200 characters for MEMORY.md and 1,375 for USER.md. They load at session start. In Hermes they are often frozen for the rest of the session so the prompt cache stays stable. Writes hit disk immediately. They do not always appear in this session’s system prompt. OpenClaw is more file-and-search oriented: • MEMORY.md for curated long-term facts in the private main session • memory/YYYY-MM-DD.md for daily working notes • Today and yesterday often auto-load. Older days stay searchable. • memory_search and memory_get retrieve chunks instead of injecting the whole archive OpenClaw also has a bundled memory-wiki plugin. It compiles durable knowledge into a structured wiki beside the active memory plugin. It does not replace MEMORY.md. What belongs in agent memory: • Who you are and how the agent should talk to you • Standing rules • Active project names and pointers to vault pages • Hard constraints, tools of record, current priorities • Corrections the agent keeps getting wrong What does not belong there: • Research dumps • Meeting transcripts • Every person you ever mentioned • Full architecture notes • A growing diary of “today we also…” MEMORY.md is an index and a constitution. It is not a library. Skills sit next to this as procedural memory: how to do a job again, not the story of the last time you did it. Session databases are episodic memory. The agent can search “did we debug this last August?” without keeping that story in the prompt. 2. Memory vault / second brain This is the library, the wiki, and the filing cabinet. Names you will see: • Obsidian vault • LLM wiki: immutable sources, LLM-owned wiki pages, and a schema file • OpenClaw memory-wiki • Agent-scoped markdown trees for knowledge, entities, and decisions The point is not “more files.” The point is compiled, inspectable knowledge. • Human-readable markdown • Stable page names • Backlinks instead of a mystery vector blob • Git-diffable • Survives swapping Hermes for OpenClaw, Claude Code, Cursor, or a local model • You can open it and see what the agent believes A vault is warm and cold memory. The agent opens a page when a pointer or a search says it matters. It does not swallow the vault at boot. Karpathy’s LLM-wiki idea is the right mental model. Do not re-derive the same answer from raw notes every session. Maintain pages. Query the index. Lint contradictions. Keep sources immutable and let the model own the wiki layer. Official Hermes guidance is blunt. Obsidian is a reviewable Markdown layer, not a replacement for MEMORY.md and USER.md. Durable facts stay in agent memory. The vault holds research, summaries, project context, and review queues. Why the mix-up happens Both look like markdown. Both get called memory. Both can be written by the agent. So people do this: 1. The agent learns something useful 2. They paste it into MEMORY.md 3. The file grows 4. Consolidation or truncation kicks in 5. Early context gets mashed into a lossy summary 6. The agent is now sure of a compressed half-truth Or the opposite: 1. They put everything in Obsidian 2. Nothing is in the hot layer 3. After compaction, standing instructions vanish 4. The agent forgets the house rules mid-session Neither layer is broken. The routing is. The hot layer answers: who am I talking to, what are the rules, and what is in flight right now? The vault answers: what do we already know about this person, repo, decision, or domain? Why you want a vault anyway MEMORY.md will never be your second brain. It cannot be. Context windows, cache, and consolidation will not allow it. You want a vault because knowledge should compound, not evaporate. Session logs are searchable. They are not a maintained model of your world. A wiki page can absorb ten sessions without forcing those ten sessions into the next prompt. You need to see what the agent believes. If the agent is going to act on a fact, you should be able to open the page, edit it, and leave a note. Agents get swapped. Vaults should not. Hermes today, OpenClaw tomorrow, Claude Code on a repo, a local model on another box. A vault on disk is the portable substrate. A hidden home-directory memory file is not. Retrieval beats stuffing. A 200-page vault in the prompt is not more memory. It is worse attention. Pointers plus a file read or a wiki search beat a stuffed system prompt every time. You become the editor-in-chief. The agent drafts. You accept, correct, or kill a page. That review loop is how you stop silent drift. Daily notes and durable knowledge are different jobs. OpenClaw already splits this: daily files for running context, MEMORY.md for distilled facts. A vault is the third hop: distilled knowledge pages that outlive both. The integration pattern that works Do not replace agent memory with a vault. Do not replace a vault with agent memory. Make MEMORY.md a router. Hot: MEMORY.md, USER.md, SOUL.md. Standing rules, identity, active work, and pointers into the vault. Warm: vault pages the agent opens on demand. Projects, people, decisions, how-tos, research. Episodic: session DB and daily notes. What happened Tuesday. Procedural: skills. How we do this job. A good MEMORY.md looks like this: Standing rules • Ask before any irreversible action. • Never put secrets in the vault. • Prefer editing existing wiki pages over creating duplicates. Active work • Project A details live in wiki/projects/project-a.md • Project B details live in wiki/projects/project-b.md How to recall • People and firms: wiki/entities/ • Decisions: wiki/decisions/ • Search the vault before answering from memory. • If a fact will matter for 30-plus days, write a vault page, then add a one-line pointer here. That last line is the whole system. How to set it up Step 0. Scope it. Do not point Hermes or OpenClaw at your entire personal vault on day one. Create a dedicated vault or folder: AgentVault • index.md • wiki/concepts • wiki/entities • wiki/projects • wiki/decisions • wiki/how-tos • daily • inbox • raw, optional immutable sources • log.md Keep journals, passwords, family notes, and client secrets out of it unless you have a hard reason and a separate scoped profile. Start with an Agent Memory folder, then expand after a clean smoke test. Step 1. Decide ownership. You own raw sources if you use the LLM-wiki pattern. Sources stay immutable. The agent owns the wiki. It writes and updates pages. You review. Both own index.md and MEMORY.md. Those stay short. If you already live in Obsidian, use a separate vault or a top-level folder. Separate is cleaner for write-happy agents. Step 2. Wire Hermes. Community path that shipped with Hermes v0.14: hermes memory setup –provider obsidian –path ~/AgentVault hermes memory status Optional but useful: • Obsidian Local REST API on localhost for live read/write during a run • An Obsidian MCP server so any MCP client shares the same brain Then tell Hermes, in MEMORY.md or a skill, the folder contract: where to write, how to name pages, and when to promote a fact into hot memory. Smoke test: ask it to write a one-page summary of this session to a daily note and add a pointer in index.md. Open Obsidian. If you cannot find and edit that note, the integration is theater. Step 3. Wire OpenClaw. Keep native files. Durable facts go to MEMORY.md. Running context goes to memory/YYYY-MM-DD.md. Then add the vault as an extra search path and enable memory-wiki if you want compiled pages and wiki tools. Let any session-memory plugin handle capture and recall. Let the vault handle compiled pages. They stack. They should not fight over the same job. Step 4. Give the agent a filing skill. This is the piece most people skip. The vault stays junk without rules. 1. Before answering a recurring question, search the vault. 2. After a decision, write or update a decision page with date, context, decision, alternatives, and links. 3. After meeting a person or vendor, update an entity page. 4. Do not duplicate. Merge into the existing page. 5. Promote to MEMORY.md only if it should shape every future session. One line plus a path. 6. Daily note equals timeline. Wiki page equals knowledge. Never invert those. 7. If two pages disagree, flag it in the log instead of silently overwriting. That is ingest, query, and lint, applied to an agent workspace. Step 5. Close the human loop. Once a week: • Skim the daily notes • Accept or rewrite new wiki pages • Delete duplicates • Pull 3 to 10 lines of truly durable truth into MEMORY.md • Demote stale hot-memory lines back to the vault If you never open the vault, you do not have a second brain. You have an agent writing fanfic about you in a folder. Git the vault. Treat agent writes like a pull request. Diffs are the cheapest safety system in this stack. What integrated actually feels like A well-wired session looks like this: 1. The agent boots. Hot memory says who you are, the rules, and three active pointers. 2. You ask about a project. The agent reads the project page instead of reconstructing it from chat logs. 3. It does the work. 4. It appends the daily note. 5. If a decision changed, it updates the decision page. 6. If the change is now standing policy, it adds one line to MEMORY.md. 7. You glance at the page later and fix the one sentence it overstated. The agent gets continuity. You keep ownership. The prompt stays small. The knowledge base gets richer. That is the whole payoff. Failure modes to avoid Whole-life vault access. Family journals and API keys do not belong in the same graph the agent can rewrite. Vault as prompt. If you find yourself concatenating the vault into the system prompt, you rebuilt MEMORY.md badly. Two sources of truth. Do not keep the same preference in USER.md, the vault, and a memory plugin with three different dates. Pick a home. Point the others at it. No promotion rule. If everything stays in daily notes, you have a diary, not a brain. No demotion rule. If everything stays in MEMORY.md, consolidation will eat your history. Unreviewed agent writes. Markdown is only safe because you can read it. Read it. The short version MEMORY.md is the agent’s working memory. The vault is the agent’s library. You need the first so the agent does not forget the rules after compaction. You need the second so the agent can accumulate a world model without setting that world model on fire every session. Set up a scoped markdown vault. Make hot memory a constitution plus an index. Teach the agent to search, write pages, and promote only what must stay hot. Review the pages yourself. That is how Hermes, OpenClaw, and the rest of the autonomous stack stop being impressive chat logs and start becoming a system that actually remembers.
