I have been running local models for a long time, got sick couple months ago of managing the scripts, and cobbled together a makeshift shell launcher that combined them all in one place. This turned out to be quite useful, but after a month i had already 500+ profiles stored in it, so i've started tweaking it here and there, and over last couple months landed on that thing below. It combines all the available local inference backends (from single machine or whatever you connect it to in lan), gives access to managing them through web panel, and most importantly - allows me to just ask agent to switch the backends on and off, as they are needed, without explaining what is where and on what port it's supposed to be on. https://preview.redd.it/uasrysnwrqth1.jpg?width=1600&format=pjpg&auto=webp&s=bb4512d5ab032f7679e9b388cbdbe300c9a4b9e3 **It is not** a runtime or a model zoo. It ships no servers and no weights. Besides your own servers and the KLIF machines you add, the only host it contacts is huggingface.co, and only when you ask it to download a model. It can help suggest You a model based on your hardware, You can click to download it, and the -cli has some features that will help Your agent benchmark and calibrate the models, but, let me repeat once more - KLIF ships no backend servers, nor any models. It's a frontend manager. Imagine library like Steam, but for local servers. Or just imagine winamp, doing inference visualization instead of visualizing the music that plays. Also, it has all the essential larping features, prefill/generations speed records, fancy animated skins, and is made in Rust to hog the least amount of resources while larping commences. I have no idea whether anyone will need this, but that's what i use now every day for any kind of local model. If You prefer running your servers manually, from terminal, from your own launcher - great, this is for people that prefer otherwise. GitHub: https://github.com/koksny/klif Video: https://www.youtube.com/watch?v=MAE393AL5As   submitted by   /u/Koksny [link]   [comments]