Shouzhuo Tech logoShouzhuo Tech

Blog

The Best SFT Data Is the Data That Fits the Model

#LLM#post-training#SFT#data-selection

In the previous post, I mentioned that there is no single best SFT loss — only the SFT loss that best fits the model. The same holds for data: the SFT data that fits the model is the best.

GRAPE: Selecting Responses That Fit the Model

The Best Instruction-Tuning Data are Those That Fit (NeurIPS 2025) proposes GRAPE:

Figure 3: An overview of GRAPE — it takes multiple off-the-shelf datasets, gathers responses for overlapping instructions, and selects responses that align with the base model's distribution for standard SFT

Assume that for a given prompt, there are multiple candidate responses. GRAPE's idea is to select the response with the lowest PPL under the model being fine-tuned, and use it as the SFT sample for training.

Formally, for each instruction, GRAPE picks the response with the highest length-normalized probability under the target model:

y⋆=argmax⁡y  1N∑t=1Nlog⁡Pθ(yt∣x, y<t)y^{\star} = \operatorname{argmax}_{y} \; \frac{1}{N} \sum_{t=1}^{N} \log P_{\theta}\big( y_t \mid x,\, y_{<t} \big)

Note that PθP_{\theta} denotes the base model — in the paper's experiments, the model being fine-tuned is always a base model, i.e., an LLM without post-training. Therefore, the PPL is computed using a template like:

Question: {instruction}
Answer: {response}

Experimental Results

Table 2: GRAPE on the Tulu-Olmo collection — GRAPE achieves the best average performance across LLaMA-3.1-8B, Mistral-7B, and Qwen2.5-7B, beating SFT on the full dataset and random selection

  1. With GRAPE, the results are better than SFT on the full dataset, and better than randomly picking a response;

  2. Using reverse GRAPE (picking the response with the highest PPL, the "Highest" setting in the table above) performs even worse than random selection (see the table above). This shows that the magnitude of PPL genuinely affects fine-tuning performance;

  3. If the multiple candidate responses come from the same generator, GRAPE is still effective;

  4. However, if the responses are self-generated by the target model itself, the performance drops, which may be related to distributional collapse;

    Table 4: Performance degradation from self-distillation on UI — self-distilled responses perform worse than the original UI responses across Mistral-7B, Llama3.1-8B, and Llama3.2-3B

  5. GRAPE also works for CoT training.

    Table 9 (paper appendix B.3): Performance metrics on the MATH dataset — with long chain-of-thought distillation, the lowest-perplexity selection (GRAPE) outperforms the highest-perplexity selection on both Qwen2.5-1.5B and Qwen2.5-3B

Open Questions

GRAPE is a prior-leaning method (reinforcing the model's existing prior), and both the training and test domains in the paper are fairly common ones. Whether GRAPE still brings gains on domains where the model's prior is weak remains an open question.

Meanwhile, GRAPE is applied to sample selection. Would combining it with a prior-leaning SFT loss (such as DFT) further improve the results? If DFT is used, is GRAPE still needed? And does GRAPE remain effective on instruct models? The paper does not explore these directions, but they are worth studying.

Two Takeaways

From this paper, two broader points can be drawn:

  1. The quality of SFT data is not only absolute, but also relative (i.e., whether it fits the model);
  2. Directly distilling from the strongest LLM is not necessarily optimal, because its data may not be the best fit for the model.

References

[1] The Best Instruction-Tuning Data Are Those That Fit (NeurIPS 2025) — arxiv.org/abs/2502.04194

[2] There Is No Single Best SFT Loss

[3] Understanding SFT from an RL Perspective

[4] GEM: Preserving Diversity During SFT