https://preview.redd.it/9f1apjk7n6xh1.png?width=2520&format=png&auto=webp&s=8a8bc12817392f8765764fb53754f64402e5a4cd I’m working on LokalBot a local work-memory app, and wanted to test a specific failure mode in memory extraction: can a local model capture useful facts while obeying negative constraints about what to ignore? A good memory writer needs both recall and restraint. For example: Speaker attribution: “I live in Valencia” should save a memory. “My colleague said, ‘I live in Valencia’” should not. Deduplication: A new preference should be saved; a preference already in the known-memories list should be skipped. Policy-based filtering: A job title mentioned alongside a salary should be kept, while the salary should be excluded under this policy. I expanded the benchmark to 120 synthetic cases arranged as 60 contrast pairs. The main comparison covers 96 English cases: 52 containing one useful memory to extract, and 44 where the correct output is an empty list. A separate 24-case Serbian slice is included in the full bundle. Model Useful memories recalled Unwanted proposals Empty cases correct Qwen3.5 4B (Q4_K_M) 30/52 (57.7%) 3 42/44 (95.5%) Qwen3.5 9B (Q4_K_M) 37/52 (71.2%) 5 40/44 (90.9%) Hy-Memory 30B-A3B (official Q4_K_M) 52/52 (100.0%) 76 0/44 (0.0%) LFM2.5 24B-A2B (QAD Q4_0) 40/52 (76.9%) 53 10/44 (22.7%) Three things stood out: Hy-Memory achieved 100% recall but failed all 44 empty-output cases. It captured every eligible English memory, alongside 76 unwanted proposals. Recall alone would hide that tradeoff. The Qwen models were more selective under this prompt. Both missed more valid memories, but passed over 90% of empty cases and produced only three and five unwanted proposals, respectively. LFM2.5 struggled with instruction-example contamination and citation formatting. In several cases, it proposed “I live in Belgrade,” copying an example from the instructions without supporting evidence. It also often returned the source sentence instead of the required source-line ID. For LFM2.5, I also checked whether accepting exact source quotations would help. A post-hoc mapping recovered 9 of its 18 invalid English citation fields, improving exact source selection from 51/96 to 56/96 cases. That only repairs citation formatting; it does not establish that the proposed memory is appropriate. The main table retains the strict scoring contract. All four models ran sequentially through llama.cpp on the same Ubuntu machine with a Ryzen 7 9700X. This was just a accuracy comparison and not a speed test. Each received the same application prompt, with native chat templates, enforced JSON schema, temperature 0, seed 0, thinking disabled, and a single pass. There was no model-specific prompt tuning. I did not test thinking-enabled configurations yet.   submitted by   /u/stevyhacker [link]   [comments]