We’ve been working on OpenEBFM (Energy-Based Foundation Models), and I wanted to share the project here and get feedback from the community. Most language foundation models today are based on directly modeling token probabilities. Our question was: Can energy-based learning scale to general-purpose foundation models? The main idea behind EBFM is to separate generation into two steps: Energy steering: given the current context, the model optimizes a latent state toward a lower-energy region using the gradient of a learned energy function. Token generation: a generative head then uses this energy-steered latent state together with the context to predict the next token. The energy and generative heads share the same Transformer backbone. During pretraining, we also parallelize energy steering across all token positions with redesigned attention masks and positional indices, so that the model can be trained at sequence level rather than steering every position separately. What we found We pretrained EBFMs on FineWeb, DCLM, and ClimbMix, and studied scaling with both training tokens and model parameters. A few results we found interesting: Data scaling: EBFMs achieve lower validation BPB than the matched GPT baselines in our experiments, and the gap generally increases with the training-token budget. Parameter scaling: from roughly 268M to 1B parameters, EBFMs improve faster than the corresponding GPT baselines. Compute scaling: EBFMs exhibit the expected U-shaped IsoFLOP curves, with the compute-optimal model size shifting upward as the compute budget grows. Generalization: we evaluated on 19 zero-shot QA tasks covering reasoning, knowledge, language understanding, and commonsense reasoning, as well as 7 cross-domain generation datasets. The same trend also persists during supervised fine-tuning. For example, on DCLM, the largest EBFM reaches an average zero-shot QA accuracy of 44.71%, compared with 39.79% for the corresponding GPT baseline in our setup. We don't think these experiments establish that energy-based models are universally better than autoregressive models—the current scale is still around 1B parameters—but they suggest that energy-based learning continues to benefit from increasing data and model scale, which we think is an interesting result given that EBMs have historically been studied mostly in specialized settings. Everything is open source We released the full training and evaluation stack, pretrained checkpoints, training recipes, and an EBFM-based chatbot so that the results can be reproduced and extended. Paper: https://arxiv.org/abs/2610.33005 Code: https://github.com/datamllab/openebfm Models: https://huggingface.co/collections/guan-wang/openebfm Paper with Code: https://paperswithcode.co/paper/2610.33005 Disclosure: I'm one of the authors of this work. We’d especially appreciate feedback on the architecture and scaling results. More broadly, I’m curious what people here think: could energy-guided generation become a practical alternative to conventional autoregressive generation as models scale further? Criticism, reproduction attempts, issues, and contributions are all very welcome! Processing img jv81qyxgijyh1... Processing img 5rbyt7ygijyh1... Processing img jf6s09yqijyh1... Processing img ufjmxdtrijyh1...   submitted by   /u/Dear_Treat3688 [link]   [comments]