I've started a series where I take a question I have about prompting or LLMs, find the research paper that tested it, and re-run that test on 2026 models. Questions like: does writing "You are an expert" actually help? Does an LLM judge prefer whichever answer comes first? Does a judge prefer a markdown answer over the same answer in plain text? Papers answered these on 2023–2024 models, and I wanted to know if the answers still hold. This one re-runs NoLiMa (2025), a needle-in-a-haystack test where the question shares no words with the hidden sentence. The needle says "Mandy lives next to the Semper Opera House", the question asks "Which character has been to Dresden?", and the model has to connect the two itself. I ran it on six 2026 models with 7,500 calls that cost $19.02 (plus $15.16 for the modified prompt re-runs). What I found: The model answer correct when context is 1k token. At 32K tokens, 9 of 10 setups scored below 85% of their 1K score. I thought NoLiMa's prompt (Use the information provided in the book snippet) caused that. So I repeated the whole experiment twice, one after removing the prompt line and one after modifying it. But the result did not change. When the question repeated the needle's words, all 10 setups scored 100% at 32K. Length alone isn't the problem; length plus no shared words is. I found out if an LLM is reasoning more, its early warning that it will provide a wrong answer. Takeaway for RAG: phrase your question in the document's own wording when you can. Full write-up: https://talhaanwar.com/blog/needle-in-a-haystack/ Code and data: https://github.com/talhaanwarch/article-experiments/tree/main/needle-in-a-haystack   submitted by   /u/mrtac96 [link]   [comments]