I'm testing LLMs to see how well they know Polish grammar. Why? Don't ask. Maybe I have too much free time, or maybe I'm some kind of freak who doesn't know what he's doing. ;) For this, I use my own tool, AI Bóbr, which is designed to test LLM performance in relation to the computer they're running on. And I noticed something strange. The model's first answer is sometimes different from the next four, even though the temperature is set to 0 and the seed is always 42. A little side note here, because one user, under this same post before I deleted it, pointed out that I'm doing Sisyphus' work by dealing with this subject. I think differently. And why did I delete the post? Believe it or not, I pressed the wrong button. So here I am posting it again. Anyway, back to the subject. Sitting there feeling defeated, just like now, because my written English is poor and algorithms on a forum about LLMs flag my posts as written by an LLM, I told myself: "Man, stop wasting your life. Go make yourself some tea, turn off the computer. Live a little, go running, walk the dogs, play with your kids. In short, just leave this damn thing alone." So that's what I decided to do... And then I sat down at the computer and ran another controlled experiment: 3 models, 10 model–task pairs, 4 variants (streaming on/off × model kept loaded/unloaded), with 5 repetitions for each variant. 200 responses in total. Observations: In this sample, streaming had no effect on the answers. Some answers changed depending on whether the prompt was reused from cache. Bielik-Minitron 7B answered a question about the history of the Polish language correctly 5/5 times after unloading the model, but got it wrong 5/5 times when the prompt was already cached. What does this tell us? That with a fixed seed, if we don't control the state of the server, we can't accurately measure what we think we're measuring. What did I get out of this? I changed AI Bóbr in version 0.8.1. I added a controlled mode that unloads the model before each attempt, confirms that it has been unloaded, and records prompt-cache statistics. Now I'm preparing to run the benchmark of 11 models again under the same conditions. And maybe I'll take the dogs out for some fresh air. Tomorrow I definitely won't touch the computer, at least not until the evening. And one more thing for everyone who reads my posts: 1. Yes, I use AI to generate posts. I read them and make corrections if necessary. 2. Yes, AI translates them into Polish and English. 3. Yes, I'll keep doing it because it's convenient and fast. 4. If someone doesn't like it — well, I'm not "Prince Polo" for eng. it will be "Mars bar", so not everyone has to like me. 5. I'm here to share knowledge, not to show off the purity of my writing — even when it's generated by AI.   submitted by   /u/Full-Travel857 [link]   [comments]