I've been making changes to both llama.cpp and Strata for my own use cases using Q38FN, Q3827B and DS41F and all three have issues with their COT considering stream tokens when touching those areas of code, resulting in them breaking out of their COT or just terminating their turn. I know there is the possibility of formal protections around end of turn tokens but from what I see there is no formal way to protect an LLM when it generates text like but I see plenty of people working on inference server code using local models so people must have a solution for it, what do you do?   submitted by   /u/SilentMobius [link]   [comments]