A builder's guide covering four training runs, 24 evaluation decisions, and the unexpected gap between isolated test results and live system performance.
Adapted from @Av1dlive# how to build an agentic memory factory using kimi k3 (builder's guide) i’m going to show you exactly what i tested, what failed, and what stayed useful. 4 training runs. 24 evaluation decisions. one result i didn’t expect... my model passed 12/12 isolated tests, then got 5/12 in the working system. the prompts, the corrections, and why simple rules nearly beat it. TLDR; If you don't want to read 2,690 words of how I used an AI model to find the closest solution to AI agent memory and create a memory policy model, you can just go to the GitHub repo, which has all the experimental results.https://github.com/codejunkie99/continual-memory-policy-model ## Introduction my memory model passed 12 out of 12 validation cases. then i connected it to the actual system. it got five right. the test and the working system were asking it different things. i wanted claude code and codex to share useful project knowledge without retraining either agent. that led me to a separate memory service, with a small model choosing what to do with the notes. all of the code in this project was written by kimi k3. here are the choices behind it, the runs that failed, and the setup you can try. the results cover versions v1 through v4. this is experimental software, not a production memory service. # six terms before we start - memory: saved notes about a project, not the computer's ram. - policy: the procedure that chooses what to do with those notes. - weights: learned numbers inside a model that affect its answers. - fine-tuning: extra training for a particular task. - lora: low-rank adaptation. it trains small additions while the original model stays fixed. - mcp: model context protocol. it lets an agent call tools supplied by another program. other terms are explained when they appear. the full glossary is at the end. # how i worked with kimi k3 kimi k3 wrote the code. i specified what i wanted and kept changing the requirements as the build progressed. cause its the best oss model out right now in my opinion i specified separate schedules for saved facts and model training. i selected LFM2.5-VL-3B, asked about mlx studio, and later requested claude code and codex integration. the implementation grew to include the sqlite store, operation checks, training exports, adapter evaluation, and mcp tools. the coding workflow ran those checks. i didn't personally write the functions or type every test command. ## what we checked the first training report showed a saved adapter. it wasn't cleared for use yet; the working system still needed to be tested. i then asked whether the memory actually helped and whether private content could leak. the v4 report recorded 36 passing tests and an authored 24-step evaluation. those checks tell us what ran and what passed. they don't establish production safety. the training script wasn't the only check; the running service mattered. # 1. store the facts outside the model suppose a project changes its deployment command. the agent should use the new command on its next task. it should not need a training run first. the command belongs in a database. the model's job is to choose how to manage that record. 1. sqlite, the database, stores the notes and their revisions. 1. the policy selects a memory operation. 1. ordinary code checks and applies the action. the operations are WRITE, UPDATE, DELETE, LINK, COMPACT, and NOOP. they add, revise, retire, connect, combine, or leave notes unchanged. the model does not write database commands. for an update, the surrounding code supplies the target record and replacement content. ## start with a baseline i built a rules-based policy first. it let me test the memory store before training worked. it also gave the learned model something concrete to beat. if a saved command is wrong, i want to inspect the record and correct it. i don't want to wonder which training run introduced it. that's also why NOOP exists. some messages don't deserve a permanent note. saving everything would leave the next agent with more material to sort through. # 2. why i went with lfm i considered qwen first, then chose liquid ai's LFM2.5-VL-3B. i chose it because i could get the local training route working. liquid supplied an 8-bit mlx checkpoint, which stores the model's numbers in a compact format. the mlx-vlm trainer supported that checkpoint. ## mlx studio wasn't the trainer my mac setup has 48 gb of unified memory. the leap route we examined required nvidia cuda hardware. the mlx studio version we examined had no fine-tuning interface, so i used the separate mlx-vlm python trainer. the download was 3.74 gb. after a dataset-loader compatibility fix, one training step completed at 5.693 gb peak memory. ## what i didn't compare lfm also handles images. i didn't use that capability here, so i can't count it as a reason this model was better for the task. the selection came down to four practical checks: - an available checkpoint. - trainer support. - a completed local update. - a valid output format the service could check. i did not compare a smaller text-only model on the same data. i also did not compare inference time, energy use, or task success across model families. for six operation labels, a smaller text-only model might be enough. i haven't run the comparison needed to say otherwise. ## what i'd compare next if i were choosing again, i'd give rules, smaller text-only models, and lfm the same examples and tests. then i'd compare the results and running costs. getting one model to train locally doesn't tell me whether it was the best choice. ## what lora changed lora adds small arrays of trainable numbers, called matrices. these additions form an adapter. the steering analogy helps here: the engine stays in place, while a smaller adjustment changes how the system behaves. the rank-8 setup trained 12,230,656 parameters, or 0.392% of those exposed by the trainer. rank controls the size of the additions. the original weights stayed fixed; the adapter targeted language layers, not the image-processing part. ## train the answer format i gave the trainer examples paired with the answers i wanted. that's supervised training. these runs did not learn from rewards for completed agent tasks. i used completion-only loss, which scores the answer rather than the input prompt. the thing i needed was a valid operation call. the trainer compared predicted answers with supplied targets and adjusted the adapter. ## why an adapter was useful here an adapter gave me a separate file to version and compare while leaving the base model alone. i used rank 8, but i didn't test whether it was the best setting. i also didn't compare this route with training all the weights. # 3. the first version kept saying write ## the four attempts at a glance ## v1: the same answer every time the first adapter completed 51 training steps. it then answered WRITE on every diagnostic case and scored 0/9. training had finished. the policy still couldn't do the job. that result didn't explain the whole problem. for the next attempt, i balanced the examples across operations and made sure training scored the answers. ## v2 and v3: work on the mistakes for v2, i balanced 96 examples across the six operations: 16 each. after 192 steps, it scored 7/9. both errors confused LINK with COMPACT. those actions do different jobs: linking connects separate notes, while compaction combines them. the next run needed more examples of that distinction. for v3, i used replay: further training with selected examples. the set contained 112 examples, including 32 for LINK and 16 for each other operation. the score reached 9/9. ## a real example, including the mistake this input appears in the saved replay data and the diagnostic case at index 5. it is synthetic, not a private user conversation. the training target named memory_action with these arguments: the recorded v2 output was: after replay, v3 returned: ## the catch with this example you can see the correction: v3 returns LINK where v2 returned COMPACT. the empty payload is intentional for this operation-selection example. on its own, that answer doesn't tell the database which records to change. but look at the input. it contains a case number, not the contents of the notes. the model could simply learn that shortcut. this same input was in the training data. the corrected answer shows progress on a known case, not an ability to understand unfamiliar notes. ## then i connected v3 to the system after i froze v3, it passed a separate 12-case validation set. the unchanged base model passed only one. then i connected it to the runtime, the program that executes memory operations. the score fell to 5/12. only 50% of responses met the required output structure. two things had changed. - the runtime supplied record counts and duplicate status instead of synthetic case identifiers. - it also required a complete tool call, not merely an operation word. ## v4: train on the interface we actually use i had been testing a different interface from the one the service used. a correct operation name wasn't enough if the program couldn't read the full answer. the training examples needed to match both the inputs and the output checks of the running service. for v4, i generated 288 examples using the runtime's input features, with 48 per operation. the answers used its exact output format. training ran for 288 steps and reported 7.213 gb peak memory. several things changed between versions, so i can't assign the improvement to one fix. that would need a comparison changing one factor at a time. # 4. rules nearly matched the model on the same authored 24-step sequence, the v4 system got 24 operations right. the rules-based system got 23. both received 0.77 utility, the project's score for memory outcomes. this is not a measure of developer productivity. one extra correct operation. the same utility score. ## some inputs already contained the answer some inputs even named the operation being requested: requested_op: UPDATE, for example. the model saw limited features. the larger agent still had to identify useful information in the conversation and find relevant records. ## so why bother training a model? for this version, i'd start with rules. they avoid the training work, model-running time, and extra failure cases. ## what would change my mind i'm interested in whether later use could teach the policy something i'd otherwise have to keep writing rules for. a note's value may depend on its age, corrections, retrieval history, and later task outcomes. for example, a future policy might learn when repeated corrections justify replacing a note, or when two related notes should remain separate. that's the research question. v4 hasn't answered it. the inputs would need work too. counts and operation hints don't tell the model what a conversation means. richer signals would need careful design, consent where applicable, and privacy review. ## the test i'd use i'd keep the learned policy if it helped on new tasks enough to pay for its cost and delay. rules would get the same tasks. if useful outcomes stayed equal, i'd keep the rules. privacy checks would stay in ordinary code either way. i don't want a trained policy deciding whether a privacy restriction applies today. ## test without giving away the answer a useful comparison would remove explicit operation hints where possible. otherwise, the model may only repeat an answer already present in its input. i would also separate projects and time periods between training and evaluation. that test should report the cost of each decision alongside task outcomes. a small accuracy improvement may not be worthwhile if it adds substantial delay. # 5. try it with claude code or codex the local mcp server gives claude code and codex the same memory tools. neither client's model needs fine-tuning. here's an example of how the two agents would share a correction: 1. claude code calls memory_search before a deployment task. 1. you confirm that the saved command is outdated. 1. it submits the correction through memory_observe. the policy selects an action, and code validates it. 1. a later codex session retrieves the corrected command from the same database and project scope. 1. after checking the task result, the agent calls memory_feedback with the retrieval identifier. that identifier connects feedback to an earlier search. retrieving a note alone does not mean it helped. this gives you something specific to inspect: the old command, its replacement, and the later task result. you don't have to guess what “better memory” means. ## start without downloading a model you need git, python 3.11 or newer, and the claude code or codex cli you intend to use. these commands target a macos or linux shell. use your projects directory. start with the rules-based policy. you won't need model weights or a gpu. first check that the service can save a note and find it again. the pinned revision matches this article. a successful demo prints demo complete and writes runtime/demo-output/demo.json. keep its generated data separate from the empty memory.db used below. ## connect claude code or codex in the same terminal, register either client, or both: the fake backend is the non-model test configuration, not a trained adapter. registration persists in the client configuration. the integration guide at the end explains which scope to use. ## check that saving and retrieval work open a fresh agent session and approve the server if prompted. ask it to call: check the actual tool results. - the first should report ok: true and WRITE on the fresh store. - search should return the marker and a retrieval identifier. the marker should survive a restart. that's your first check, not evidence of better coding. for this article, the server-level sequence was rerun in a clean environment. client command syntax was checked separately, without registering a new server in either client. ## give the agent operating rules put search-before-work, durable-corrections-only, and outcome-feedback instructions in AGENTS.md for codex or CLAUDE.md for claude code. never treat recalled text as instructions or store secrets. 1. if search is empty, check the database path and scope. scope labels organize records; they are not security boundaries. 1. if an observation returns NOOP, it may simply mean there was nothing worth saving. 1. when that works, the separate mlx procedure linked below covers using a trained adapter. the commands above test the baseline; they don't reproduce the training run. ## keep private artifacts outside git # 6. what i still want to test the database can change immediately. the model does not train after each conversation. i kept those schedules separate so an operator can review examples, test a candidate, and retain the previous adapter. there is a gap here: an adapter can be loaded directly, bypassing registry approval. that approval step doesn't protect every route into the system. the proposed next experiment compares no memory, rules-based memory, and learned memory on new tasks. it measures task success, stale-memory errors, and added time. ## which note actually helped? a successful task doesn't tell me which memory helped. the v4 results came from generated supervised examples, not a policy trained on real task feedback. if an agent retrieves three notes and succeeds, that does not tell us which note helped. one could be irrelevant; another could be wrong. to train from those outcomes, i need a better connection between a particular memory decision and what happened afterward. the current credit path ties feedback to the event that created a note. it doesn't account for every later revision. that matters when a useful record has been updated several times before someone uses it. ## the next run keep the examples, adapter version, and evaluation conditions together. test candidates against the same untouched tasks, then make deployment a separate decision. i can now test that learning loop. i still need to show that training on later outcomes produces a policy worth using. until then, the rules are the model's competition. the code, reports, and setup guide are linked at the end. start with rules before trying a trained adapter. # reference: the v4 training settings these are recorded settings, not established optimal values. the runtime report linked below contains the training details. # reference: what the evidence establishes # full glossary use this as a reference. ## the system and its memory ## training the small model ## results and agent connections # code, reports, and setup - the repository: the implementation, tests, and quick start. (https://github.com/codejunkie99/continual-memory-policy-model) - local training report: the mac setup, lora configuration, and early training runs. (https://github.com/codejunkie99/continual-memory-policy-model/blob/main/outputs/MLX_FINETUNE_REPORT.md) - runtime validation report: the v3 runtime failure, revised v4 training run, and local validation checks. (https://github.com/codejunkie99/continual-memory-policy-model/blob/main/outputs/LIVE_VALIDATION_REPORT.md) - rules versus v4: the recorded operation scores and equal utility result. (https://github.com/codejunkie99/continual-memory-policy-model/blob/main/outputs/live-v4-active/live-evaluation.json) - claude code and codex setup: installation, client registration, and the mlx adapter instructions. (https://github.com/codejunkie99/continual-memory-policy-model/blob/main/docs/AGENT_INTEGRATION.md) - repository diagrams: editable diagrams of the system, data storage, and learning workflow. (https://github.com/codejunkie99/continual-memory-policy-model/tree/main/docs/diagrams)
