Hypothetically, say BigScience was to launch BLOOM 2 tomorrow, a truly open-source LLM that beats even closed-source frontier models on most benchmarks. To accomplish such strong performance, BLOOM 2 needed to incorporate several highly novel techniques that nobody's thought of before. However, since BLOOM 2 is fully open-source from end-to-end, labs like OpenAI and Anthropic can literally steal these innovations and incorporate it into their own closed-source models. OpenAI and Anthropic probably have their own innovations / quirks they build into their models, novel techniques that aren't known to the public. When combining BLOOM 2's innovation with OpenAI's closed-source techniques that no other lab has access to, the performance of OpenAI's next generation of LLMs might compound and ultimately beat BLOOM 2. In this scenario--where closed-source labs have access to novel techniques that no other lab has independently discovered--how can truly open-source models ever catch up? Whenever open-source researchers develop and publicly release an innovation, closed-source labs can simply incorporate it into their own models. But the reverse isn't true because open-source researchers can't benefit from the innovations that closed-source labs keep to themselves. As a result, closed-source labs can continually build on both their own private discoveries and the innovations of the open-source community, while open-source models only have access to the latter. Wouldn't this allow closed-source labs to literally always be ahead?? How can open-source models ever catch up? Is there something I'm missing? Would really appreciate some input   submitted by   /u/CringeyAppple [link]   [comments]