Hey guys, I threw together a small Python script called llm-drift-bench to help figure out what actually breaks when you switch to a new model or drop to a lower quant. Basically, you just feed it saved responses from your old model and the new one, and it spits out the exact differences - like dropped JSON keys, messed up numbers, or text changes. (The attached video is just a dummy example showing it catch a missing field and a changed invoice total, not real Q4 vs FP16 data). Since it just runs on saved JSON/text outputs, you don't even need a GPU or API keys to test it out. I just pushed v0.1.0 and I'm working on the next update. Repo is here: https://github.com/anant-goel/llm-drift-bench https://i.redd.it/ckjzqvyqae1i1.gif Question for you all: what specific things usually break for your use cases when you swap models? I want to build out some test cases based on actual problems people are running into.   submitted by   /u/Worldly-Knowledge-90 [link]   [comments]