I've been working on improving llama.cpp support and performance on older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures. The goal is simple: get more life out of older CUDA hardware instead of treating it as e-waste. Achieved Performance On MTP Qwen3.6-35B-A3B with Q8_0 quantization, I've observed the following (generation) speeds using 4x Tesla K40c GPUs (Kepler, compute capability 3.5): Previously unable to start → 35 tps Fixes to cuBLAS compute-type selection for legacy gpu in mainline llama cpp to run on hardware that previously failed during initialization (cc35-50). Approximately 35 → 55 tps after enabling CUDA graphs on my four-K40c setup, under the same reported clock configuration. ~5–10% end-to-end improvement: testing with Windows NCCL integration compared with the previous configuration. Throughput reaching above 60 tps while P.P. speeds are variable between 70-100 tps in my testing. These are preliminary results from my own system, not universal benchmarks. More systematic testing across architectures and workloads is still needed. I'm especially interested in hearing from people running Kepler, Maxwell, Pascal, Volta, and other older NVIDIA GPUs. If you have legacy hardware, I'd love to compare results and test compatibility across different configurations! Project / source: https://github.com/theIvanR/llama-on-legacy-gpu Llama.cpp pull request: https://github.com/ggml-org/llama.cpp/pull/30727 Windows NCCL fork: https://github.com/theIvanR/nccl-on-windows Feedback, reproducible benchmarks, and contributions are very welcome!   submitted by   /u/TheSpicyBoi123 [link]   [comments]