I have gone through a series of experiments related to an interesting project where I try to teach a local llm a new domain through continued pretraining (CPT). The different experiments are spread across the four phases below: Phase1 talks about how to teach an llm a new domain through CPT and picking a training set that will generalize well to unseen questions Phase 2 does a comparison between the performance of reasoning across internalized knowledge (CPT) vs. RAG injected content Phase 3 takes a more practical approach where the CPT trained knowledge is enriched by combining it with RAG instead of viewing the two approaches as competing solutions The final part shows the comprehensive eval strategy used to measure performance during the project. Among other things, this involved SFT fine tuning of the CPT trained model to teach it to output responses based on a strict schema instead of English sentences. The schema approach is used to simplify strict eval checks. The local model used for this project is qwen 3.5 4B. Unsloth was used for both CPT and SFT LORA training. I have provided a summary of my findings here in case someone is interested in reading more about it: https://www.teachmecoolstuff.com/viewarticle/domain-specific-training-and-fine-tuning-of-an-llm   submitted by   /u/funJS [link]   [comments]