Just a quick PSA. llama.cpp does have prompt caching. if you are running large context lengths and have long multiturn projects, increasing -cram can provide you with massive speedups. There is a point where context lengths can get so large that 8192mb is not enough and the whole context needs to be re processed again. personally, I have found 20480 to work well with Qwen 27B 3.8 at 262K context. the main downside is this uses more ram. vram usage doesnt increase.   submitted by   /u/My_Unbiased_Opinion [link]   [comments]