Shouzhuo Tech logoShouzhuo Tech

Blog

Accelerating GRPO with Down-Sampling

#LLM#reinforcement-learning#post-training#GRPO

Paper: Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning.

A side-by-side Chinese-English translation of the paper is available here: paper-translator.

Figure 2 of the paper: visualization of three training strategies — vanilla GRPO trains on all n rollouts and leaves inference hardware underutilized, GRPO-GA adds gradient accumulation at the cost of more sequential policy-update steps, while GRPO-PODS trains on only m carefully selected rollouts, maximizing inference utilization and yielding better performance

  1. The paper proposes that during the policy update phase of RL, one can apply down-sampling — keeping only a subset of rollouts to participate in the policy update. As validated, max-variance downsampling (keeping the subset with the largest reward variance) is an effective down-sampling strategy for GRPO: compared with vanilla GRPO without sampling, it speeds up training by at least 1.7×, with performance on par or even better;

Figure 3: Performance and per-step runtime comparison of standard GRPO and GRPO-PODS with max-variance down-sampling across datasets and hardware — PODS reaches equal or better test accuracy in far less training time

  1. The paper provides an algorithm with O(n log n) complexity that quickly finds the max-variance subset of the rollout set;

Algorithm 2 of the paper: Max-Variance Down-Sampling — sort rollouts by reward, then greedily grow the candidate subset from both ends whenever it increases the reward variance

  1. The paper experiments across various rollout sizes (n) and subset sizes (m), and finds the effect of max-variance downsampling is quite robust.

    Figure 4 of the paper: GRPO-PODS with max-variance down-sampling stays robust across rollout sizes n and update sizes m — accuracy curves nearly overlap while per-step time scales with n and drops as m grows

    The authors suggest setting the down-sampling ratio between 2 and 4 in practice;

  2. Down-sampling is suitable when the policy update becomes the performance or memory bottleneck. One scenario is a large rollout size;

    Figure 1 of the paper: inference scales efficiently while policy updates become memory-bound in RLVR — fine-tuning Qwen2.5-3B on GSM8K with 8 A100 GPUs, policy updates hit OOM beyond 32 rollouts per GPU and fall back to much slower gradient accumulation, while per-token inference time keeps dropping with batching

    another is long contexts (e.g., Agent scenarios — see the figure below);

Figure 6 of the paper: wall-clock training time breakdown for Qwen3-4B on multi-turn tool-calling and math reasoning tasks — policy update time (orange) dominates rollout generation (blue) and grows rapidly with the number of rollouts, revealing a severe computational asymmetry

  1. Note, however, that all experiments in the paper are based on the GRPO algorithm; whether down-sampling works for PPO and other RL algorithms remains unknown.

References

[1] Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning (TMLR 2026)

[2] Paper translator (side-by-side Chinese-English translation)