Repository navigation
can not reproduce the SWE-bench verified performance #5
Description
https://huggingface.co/datasets/GAIR/OpenSWE/blob/main/openswe_traj.jsonl
Using this carefully curated 13k trajectory and the ms-swift training framework, we are unable to reproduce the performance reported in the paper.
We are unsure whether the issue lies with the dataset or the framework, even though we have made every effort to align the hyperparameters with those specified in the paper.
After completing 3 epochs, the evaluation score is extremely low, whereas the paper mentions that a score of 50+ can be achieved after 3 epochs (approximately 200 to 300 steps).
Could you please make the training scripts and code publicly available? Thank you very much.
Activity
Fu-Dayuan commentedon Aug 28, 2026
We use slime to train our model. (a little bit modified, but same feature slime already support now.)
For inference:
The temperature should be 0.6 and repeat penalty should be 1.2 （maybe 1.1 is ok？）, docker(acs) 4cpu32G, sglang 0.4.9 (new sglang is ok),
We found if you use temp=0, the score will be really bad (6-10 score lower)