Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning
Zilin Xiao, Qi Ma, Chun-cheng Jason Chen, Xintao Chen, Avinash Atreya, Hanjie Chen, Vicente Ordonez
RA-RFT reframes retrieval for reasoning tasks: instead of pulling in documents that look similar, it trains a retriever to surface problems that share the same underlying reasoning pattern, even if superficially different. The retriever is trained via gold-relevance distillation to rank by expected reasoning benefit, and the policy model is then fine-tuned with reinforcement learning using those analogous demonstrations as scaffolds. On AIME 2025, this approach lifts average@32 accuracy by 7.1 points over GRPO for Qwen3-1.7B and 2.8 points for Qwen3-4B — meaningful gains on a benchmark where incremental improvements are hard.
No production traction yet. The GitHub repos referencing this paper are all arxiv aggregators and paper-tracking lists, not implementations. Zero citations on Semantic Scholar as of now. The idea is directly applicable to math tutoring systems, code generation, and any reasoning pipeline already running RAG, but nothing is shipping from this work yet.
Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern. We propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap, and then fine-tunes the policy model via reinforcement fine-tuning methods with retrieved analogous demonstrations, so the model learns to leverage reasoning traces under verifiable outcome rewards. We further analyze the diversity of retrieved contexts and find that reasoning-aware retrieval surfaces complementary solution strategies that provide distinct reasoning scaffolds for individual problems. Across challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning methods. For example, it improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively -- suggesting that reasoning-aware retrieval is a complementary axis of improvement and orthogonal to advances in reward design or training curricula.