I stumbled on an open source local inference app called Magnitude. I couldn't find any earlier posts about it on r/LocalLLaMA, although there is one post with almost no comments on r/LocalLLM . It claims to be significantly faster (2x) than Ollama and llama.cpp because it "tunes its kernels on your hardware to find the fastest way to run each model". The install includes both a desktop app and CLI tools. I got curious so I tried it on two of my machines: a laptop with a RTX 3060 Mobile (6GB VRAM) GPU + 24GB RAM running Linux Mint, and on a headless Raspberry Pi 8 8GB running Raspberry Pi OS (Debian). On both machines I installed the .deb packages (amd64 for the laptop, arm64 for the RPi). On the laptop I used the GUI, on the RPi I used the CLI. First impressions Magnitude takes a long time to start up (especially the desktop app) and uses lots of CPU. Apparently there is some calibration process going on where it figures out my hardware. It wasn't always obvious what is happening, you just had to wait and see. It wants to download models directly from HF Hub. It will also auto-discover GGUF files from some locations, such as HF, llama.cpp and Ollama cache directories. For many of my models it found in those, it said it cannot run them. So model support is limited to a few dozen models, but those include Qwen, Gemma, LFM and MiniCPM families so the most common ones are covered. When Magnitude downloads models, they will be saved under ~/.magnitude/models/ . This tripped me up on both machines because my home directories are relatively slow: on the laptop /home is encrypted and on the RPi it's on a SD card. On both machines, I have a separate data directory where I keep my GGUFs. I solved this by symlinking, so the GGUFs actually end up in the faster directory. On my laptop, it will only suggest small models that fit entirely in VRAM. It includes a catalog with a longer list of models, but all the larger MoEs (e.g. Qwen3.6-35B-A3B) are listed as "doesn't fit this machine". Apparently it cannot do partial CPU offload? This is a pity, because I would like to run e.g. Tiel-Coder as a coding assistant and it works quite well with llama.cpp partially on CPU, but a speedup is always welcome. It works quite similar to Ollama. There is a main server process that needs to run in the background (the desktop app starts it up automatically) and then you can use the GUI or CLI to download new models as well as load and unload them. There is a chat UI (both in-app and browser) and an OpenAI-style API at http://localhost:10100/inference/v1 . Benchmarks I decided to download and test the new LFM3-1.7B model (Q8) on both machines and compare against latest llama.cpp. The model should fit in 6GB VRAM or 8GB RAM with full 256k context. Magnitude overestimated its VRAM usage: it says 5.3GB, but actually only around 4.2GB was reserved. This probably also means that some models it said are too large would actually have fit in VRAM. I used llama-benchy to benchmark both Magnitude as well as llama.cpp running the same GGUF to get a fair comparison of PP/TG speeds. I didn't do any manual tuning for either this time; with llama-cpp, I only used llama serve with the -m parameter pointing to the GGUF. Laptop, 3060 Mobile GPU benchmark command: uvx llama-benchy --pp 2048 --tg 512 --base-url http://localhost:10100/inference/v1 --model LiquidAI/LFM3-1.7B --served-model-name lfm3-1.7b:gguf:q8 Magnitude results: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------------|-------:|----------------:|-------------:|--------------:|---------------:|----------------:| | LiquidAI/LFM3-1.7B | pp2048 | 5257.71 ± 81.66 | | 406.02 ± 6.11 | 389.62 ± 6.11 | 406.02 ± 6.11 | | LiquidAI/LFM3-1.7B | tg512 | 91.26 ± 0.24 | 91.67 ± 0.47 | | | | llama.cpp results: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------------|-------:|----------------:|--------------:|--------------:|---------------:|----------------:| | LiquidAI/LFM3-1.7B | pp2048 | 5710.16 ± 16.44 | | 359.44 ± 1.04 | 358.60 ± 1.04 | 359.44 ± 1.04 | | LiquidAI/LFM3-1.7B | tg512 | 112.29 ± 0.08 | 113.00 ± 0.00 | | | | Verdict: llama.cpp wins. Magnitude was 8% slower in PP and 19% slower in TG. Raspberry Pi 5, CPU-only benchmark command: uvx llama-benchy --pp 128 --tg 128 --base-url http://localhost:10100/inference/v1 --model LiquidAI/LFM3-1.7B --served-model-name lfm3-1.7b:gguf:q8 Magnitude results: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------------|-------:|------------:|------------:|------------------:|------------------:|------------------:| | LiquidAI/LFM3-1.7B | pp128 | 5.07 ± 0.01 | | 25132.36 ± 145.29 | 25115.54 ± 145.29 | 25132.36 ± 145.29 | | LiquidAI/LFM3-1.7B | tg128 | 3.59 ± 0.02 | 4.00 ± 0.00 | | | | llama.cpp results: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------------|-------:|-------------:|------------:|-----------------:|-----------------:|-----------------:| | LiquidAI/LFM3-1.7B | pp128 | 31.42 ± 1.16 | | 4082.18 ± 154.95 | 4079.61 ± 154.95 | 4082.18 ± 154.95 | | LiquidAI/LFM3-1.7B | tg128 | 5.25 ± 0.14 | 6.00 ± 0.00 | | | | Verdict: llama.cpp wins by a large margin. Magnitude was 84% slower (!) in PP and 32% slower in TG. Conclusion It's interesting to have a new general purpose (not model-specific) inference tool available that is not based on llama.cpp (Magnitude is mostly written in Rust and TypeScript). However, I couldn't verify their claims about being faster than Ollama or llama.cpp; at least for my hardware (3060 Mobile and Raspberry Pi CPU-only), Magnitude was clearly slower than llama.cpp. However, this was just a quick test and it's possible that I have done something wrong or that Magnitude would have worked better on more powerful GPU hardware. Have others tried it? No AI was involved (except the LLM I benchmarked) when writing this post.   submitted by   /u/OsmanthusBloom [link]   [comments]