We fine-tuned Alec Radford’s 1930 vintage LLM to solve SWE-bench issues.
After just ‼️250‼️ training examples, the model solves its first issue, a simple patch to the xarray library.
🧵👇
Ricardo Olmedo on X: "We fine-tuned Alec Radford’s 1930 vintage LLM to solve SWE-bench issues. After just ‼️250‼️ training examples, the model solves its first issue, a simple patch to the xarray library. 🧵👇"
We scale fine-tuning to ~75k training examples, or 1B tokens. This takes the base model from 4% pass@100 on HumanEval to 4.5% ‼️pass@1‼️ on SWE-bench, a much more challenging agentic benchmark. Having pre-trained only on pre-1931 data!Surely a model pre-trained on the web would fare much better? Yes, and no. We also fine-tune their web-retrained model, and observe a modest +1% solve-rate on SWE-bench, achieving 5.7% pass@1 compared to 4.5% Surprisingly little seems to be lost by throwing away the internet.What holds the 1930 model back is that it is severely undertrained (only 260B tokens), rather than its pre-training data. I’m excited to see further development of vintage models. Which capabilities does web pre-training provide that are not easily recoverable via post-training?Check out the SWE-bench fix discovered after just 250 training examples. The fix itself is very simple, but the trajectory demonstrates the kind of agentic reasoning we’ve come to expect from modern models. 🔗 ricardodominguez.github.io/blogs/pydata__…Do you have compute to spare? We’d love to see the full scaling curves comparing the 1930 and web-pretrained models as post-training is scaled up. Models & training data 🔗 huggingface.co/collections/ri… GitHub repo 🔗 github.com/RicardoDomingu…Curious about how pre-training data affects post-SFT performance? Check out Section 4.1 of arxiv.org/pdf/2407.07890
