Continuation of https://www.reddit.com/r/LocalLLaMA/comments/1wynomx/k_llamacpp_moe_optimizations_expert_residency/ This time I finally get speeds compared to Strata for ik_llama.cpp on 6gb vram rtx 2060 ( generates about 45–48 t/s on short prompts and about 37 t/s at 28K context. Prefill runs at about 430–700 t/s ). . Idea same, implementation little different. https://github.com/IceFog72/ik_llama.cpp More info on github page. I don't even trying to merge it as i have too little feedback (merging things that fully tested only on 2060 is ... ). And it need to be broken and clean up on multiple things as there bunch of different stuff added.   submitted by   /u/IceFog72 [link]   [comments]