Shouzhuo Tech logoShouzhuo Tech

Blog

When Updating Parameters, RL Takes the Path Not Taken

#LLM#reinforcement-learning#post-training#RLVR

Paper: The Path Not Taken: RLVR Provably Learns Off the Principals.

  1. Compared with SFT, RL training updates far fewer parameters;

Table 1 of the paper: update sparsity in SFT vs. RLVR across models and algorithms — SFT leaves only 0.6%–18.8% of weights unchanged while GRPO/DAPO RL updates are far sparser at 36%–92%, confirming RL touches far fewer parameters

  1. Even more striking: the same pretrained model, fine-tuned with different RL algorithms and datasets, ends up with a high degree of overlap in its parameter update regions. This shows that RL's update trajectory is model-conditioned;

Figure 2 of the paper: consensus ratio of weight updates across five RLVR runs — the 13th layer's Q/K/V/O and MLP-down projections show lighter bands where most runs update the same coordinates, a stable stripe-like routing pattern rather than random scatter

Table 2 of the paper: cross-run stability for the 13th block — Jaccard overlap between runs (0.55–0.60) is well above the random baseline (0.37–0.47) for every projection

  1. The authors propose a Three-Gate Theory to explain both phenomena:

    • Gate I — the KL leash effect: each RL update step does not drift too far from the current policy, so the parameter changes stay small;
    • Gate II — model geometry: constrained by the KL leash, RL only optimizes in directions that do not change the model's original parameter structure too much — i.e., the "off the principals" in the paper's title;

    Figure 4 of the paper: spectral geometry under SFT vs. RLVR on Qwen3-8B — RLVR keeps a stable top-k spectrum with minimal subspace rotation across layers, while SFT rotates principal subspaces by 75° or more

    Figure 5 of the paper: RL avoids updating principal weights — compared across three models, the layer-wise overlap between RL update masks and the principal weight mask is consistently sub-random

    • Gate III — precision: the amplification effect of bf16 precision; because micro-updates get masked, the parameter updates become even sparser;

  2. Gate I explains why the parameter changes are small; Gate II further explains why RL's parameter updates are model-conditioned;

  3. Beyond RLVR tasks, RLHF, Agentic RL, and DPO/SimPO all share the off-the-principals characteristic;

Figure 8 of the paper: spectrum under RL in agent tasks — across six agentic models (multi-turn interactions and tool use), RL leaves singular-value spectra nearly unchanged and only slightly rotates the top-k subspaces

Figure 18 of the paper: spectral geometry under the RLHF setting — Llama-3-Instruct-8B checkpoints trained with DPO and SimPO also preserve layer spectra and induce only minor rotations

  1. Even though DAPO removes the KL loss, its importance ratio + clipping still give it the KL leash effect; DPO has an implicit KL regularization, so it has the KL leash effect too; SimPO has no implicit KL regularization, but its training tricks also keep the KL divergence low (see the original SimPO paper);

  2. SFT updates on the principals — the exact opposite of RL. Therefore, PEFT improvement techniques that suit SFT (such as PiSSA) are not necessarily good for RL; RL deserves PEFT techniques of its own;

Figure 1 of the paper: SFT vs. RLVR optimization geometry — SFT follows a predefined route "over the mountain" targeting principal weights, while RLVR is steered by an implicit compass around it: update masks avoid principal-weight positions and the top-k subspace rotates less, preserving the spectrum

  1. The following known phenomena echo this paper nicely: RL only needs a low-rank LoRA to approach full fine-tuning performance; RL generalizes better while SFT leans toward memorization.

References

[1] The Path Not Taken: RLVR Provably Learns Off the Principals