We should "digest" them into reusable sub-modules. Ok from the beginning: recently we are struck by this epidemic of custom engines, vibecoded solutions tailored on a very specific hardware. probably behind each one of them there's really some great idea, trick, kernel, optimization that has made the difference for its author. But many times they are so niche, they are basically useless for the majority of the community. AT the same time, fragmentation is bad for the open source community: we should pool resources not disperse them. So here's my idea, which I hope could spark something bigger. Imagine every time a custom engine gets published, some agents pick it up, dissect it into submodules, optimizations, kernels, strategies. Everything gets classified into a database, depending on the specific hardware, quantization, LLM model used. duplicate entries gets removed. code snippets gets benchmarked and assigned some scores and some tags Over time, A huge database of building blocks for inference engines gets built. From that, anyone can search filtering for the model/quant/hardware they need, and a custom inference engine is built for YOUR use case, a pool of modules/strategies can be tried and swapped until the optimal configuration it's found. Bonus idea: People could donate their free agentic time (subsidized tokens, the remaining weekly from your OpenAI / Anthropic / Z.ai subscription) to the project, and in some way we get some payback from these companies. They made our hardware expensive, we claw back some value by using their computing power to make US run locally as fast as possible. I'm not the right person to do this, but maybe someone could catch this idea and make it work. Of course there's already an immense swath of repos to start from (think all the custom flavors of llama cpps stratas, exllamas, vllms sglangs....) , this idea could work even if custom inference engines stop tomorrow.   submitted by   /u/nasone32 [link]   [comments]