Blog
Paper Breakdowns
Plain-language walkthroughs of the papers behind modern AI — for developers who want the ideas, not just the headlines.
Accelerating GRPO with Down-Sampling
PODS (Policy Optimization with Down-Sampling) keeps only an informative subset of rollouts for the policy update: max-variance down-sampling (keeping the subset with the largest reward variance) makes GRPO at least 1.7x faster with equal or better performance, with an efficient O(n log n) selection algorithm.
Jul 16, 2026When Updating Parameters, RL Takes the Path Not Taken
RLVR updates only a small fraction of parameters, and the update regions overlap across different RL algorithms and datasets — the update trajectory is model-conditioned. The paper explains this with a Three-Gate Theory (KL leash, model geometry, bf16 precision): RL learns off the principal directions in weight space, while SFT targets principal weights. Implications: SFT-era PEFT techniques may not suit RL.
May 27, 2026The Best SFT Data Is the Data That Fits the Model
GRAPE (NeurIPS 2025) selects, for each instruction, the candidate response with the lowest perplexity under the model being fine-tuned. This simple prior-leaning data selection beats SFT on the full dataset and distillation from the strongest model — showing that SFT data quality is relative to the model, not just absolute.
Apr 23, 2026There Is No Single Best SFT Loss
A parametrized family f(p) = -(p^alpha - 1)/alpha unifies SFT objectives: NLL (-log p) is prior-averse, while -p and thresholded variants are prior-leaning. The paper shows which objective wins depends on the model-capability continuum: strong prior favors prior-leaning, weak prior favors NLL. Even SFT variants like DFT, ASFT, PSFT, and GEM are not universally better than NLL.
Mar 25, 2026GEM: How to Preserve Diversity During SFT
Standard SFT with cross-entropy loss pulls every position's token distribution toward a one-hot target, hurting output diversity. GEM (Generalized Entropy Minimization) turns the all-to-one update into a sparse update, preserving diversity, improving Best-of-N scaling and downstream RL, and mitigating catastrophic forgetting.
Mar 13, 2026Understanding SFT from an RL Perspective
From the gradient perspective, SFT is a special form of policy gradient; from the loss perspective, the SFT loss is a lower bound of the RL loss. This post walks through the math connecting SFT and RL, and three SFT improvement algorithms inspired by this connection: DFT, PSFT, and ASFT.