So, when I first released ScaffoldCoT, I never thought it would reach more than a handful of people. I was happy to see multiple thousands of consistent downloads month over month since its release. Since this is where I initially posted about the project, I thought you guys would be interested in recent updates! TL;DR - Scaffold CoT is now Scaffold Chat (tool calling removed, it was causing significant tool calling reliability issues), Scaffold Agent is out (~1B tokens of real, executed coding agent sessions with the same thinking scaffold), and a heavily modified 5-10B model trained on all of this is on the way. Hi all! Quick update post on Scaffold CoT for anyone who saw the original. Changelog Renamed to Scaffold Chat. Same repo (old link redirects), load_dataset("Specific-Labs/Scaffold-CoT") still resolves, old revisions still in history. Sorry this did not happen before, but a major training run failed on tool calling benchmarks due to this, so it was back to the drawing board. Removed all tool calling. 276,389 examples gone (tool_use + tool_use_complex, literally every tool_call/tool_result event in the dataset). I found several synth gen harness issues in tool calling examples that propagated failure modes into the dataset. Everything that remains is byte identical to the previous revision. Now 3,450,159 examples, 16 domains, 75 shards, ~13GB Junk domain label tail from the original post (97 labels, 542 examples) is cleaned up. Every example has an exact domain/subdomain, counts sum to the whole dataset Added SCAFFOLD_SYSTEM_PROMPT.md to the repo. The examples were generated without system prompts, this is the format protocol distilled from the generation code for anyone who wants the scaffold conditioned on a system prompt instead of unconditional. Calibration block untouched: antihal + antihal_kb_refined + selfcheck is still ~675k examples. This has been a high impact category for general chat and knowledge bmks. New dataset: Scaffold Agent This is what the executed tool chains grew into. It consists of real Pi code traces of real coding projects from open near-frontier models, using the same scaffold system: 19,652 records from 6,792 coding sessions, 1,005M tokens (899M trainable after masks). Teacher model in a real agent harness (Pi) editing real repos at pinned commits + building from scratch, simulated user on the other side powered by another model with its own harness and instructions to ensure variety and quality Every tool call really executed, every result is real output, every action preceded by the same scaffold block (Inventory / Interaction / Execution) in reasoning_content. Fabricated tool output is structurally impossible and any such instance was cleared Sessions only kept if they actually verified (hidden tests + code quality grade on the main shard). This is a ~20% subset of the total generation that was of extremely high quality. I wanted to ensure these were top-tier coding traces. 2 LLM judge passes and green final CI were required for approval into the final dataset. Each record is one compaction segment (≤64k tokens), self contained with system prompt + tool definitions + history. Do not join segments. training_sft_config.yaml in the repo documents the whole setup Scaffold Chat teaches the format and general reasoning, Scaffold Agent teaches acting with tools inside that same format. My recommendation is to mix them with other larger datasets while ensuring there is at least one scaffold example per batch, which leads me to... Coming soon Currently training a heavily modified model in the 5-10B range on this data (base model, full SFT, and a new merge approach I have not seen applied for scaled releases yet). Results post when it is done and actually evaluated, not before. The idea is to build a small dense model that can run on a range of GPUs and laptops that is optimized for coding specifically, and will move extremely fast because of our CoT framework and the small size. This has been a massive project with several mistakes made along the way, and results are finally looking up, with scaffold datasets being the core of the training run, so I am very excited to bring it to the community soon. I know it can sometimes be a bit annoying to announce something before its available, but I couldn't contain telling the community about it lmao. Happy to answer any questions about it (or anything else for that matter) in the meantime. Links Scaffold Chat: https://huggingface.co/datasets/Specific-Labs/Scaffold-Chat Scaffold Agent: https://huggingface.co/datasets/Specific-Labs/Scaffold-Agent Both CC-BY-4.0. Genuinely hope these prove useful!   submitted by   /u/Saraozte01 [link]   [comments]