ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation

Proactive Recommender Systems (PRSs) aim to guide user preference shift toward target items by generating paths of intermediate recommendations. Reinforcement learning (RL) provides a principled framework for optimizing such sequential decision tasks, as path rewards can naturally capture both short-term acceptance and long-term guidance effectiveness. However, naively applying policy gradients to PRS results in deficient gradient estimation. We identify two deficiencies: (1) path-level rewards decompose into step-level rewards with positive mean, creating a length-dependent bias that causes gradients to favor path extension over meaningful exploration; (2) weighting each step by the entire path-level reward ignores the decomposition structure, leading to high gradient variance. To rectify these two deficiencies, we propose an effective RL framework ProRL with two novel mechanisms for proactive recommendation. First, Stepwise Reward Centering subtracts expected rewards to neutralize length-dependent bias, ensuring that path extension yields zero expected gradient signal. Second, Position-Specific Advantage Estimation leverages the reward decomposition structure to compute step-dependent baselines, reducing gradient variance. Together, these mechanisms yield policy gradients that precisely target path quality. Our experiments on three real-world datasets demonstrate that ProRL significantly outperforms state-of-the-art PRSs. Our code is available at https://github.com/hongruhou89/ProRL.

📋 한 줄 요약

**[Proactive Recommendation / RL]** ProRL이 policy gradient의 두 결함(길이 의존 bias·고분산) 해소 — Stepwise Reward Centering으로 length bias 중성화·Position-Specific Advantage Estimation으로 step별 baseline, 3개 실세계 데이터셋에서 SOTA 능가.

🎯 핵심 기여도

Proactive Recommender System(PRS)이 intermediate recommendation의 path를 생성해 사용자 선호를 target item으로 shift시키는 것을 목표로 함을 배경 소개.
강화학습이 sequential decision task 최적화의 principled framework를 제공하며 path reward가 short-term acceptance·long-term guidance effectiveness를 naturally 캡처할 수 있음을 활용.
Naïve policy gradient를 PRS에 적용 시 발생하는 두 결함 식별 — (1) path-level reward가 positive mean의 step-level reward로 분해되어 path 확장을 favor하는 length-dependent bias 생성, (2) 각 step을 entire path-level reward로 weighting하면 decomposition structure 무시·고분산.
ProRL 제안: (a) Stepwise Reward Centering이 expected reward를 빼서 length-dependent bias 중성화 — path 확장이 zero expected gradient signal을 yield하도록 보장, (b) Position-Specific Advantage Estimation이 reward decomposition 구조를 활용해 step-dependent baseline 계산·gradient variance 감소.

💡 핵심 아이디어

Proactive recommendation의 policy gradient는 path-level reward가 step-level로 분해되며 발생하는 length bias와 step별 advantage 결여로 인한 고분산이라는 구조적 결함을 가지며, 이를 두 가지 mechanism(centering + step-specific baseline)으로 정확히 rectify해 path quality에 directly target하는 gradient를 얻을 수 있다.

🔬 기술적 접근법

**방법론**: ProRL — Stepwise Reward Centering + Position-Specific Advantage Estimation.
**핵심 기법**: (1) Path-level reward의 step-level 분해 분석, (2) Stepwise Reward Centering — expected reward 빼서 length-dependent bias 제거, (3) Position-Specific Advantage Estimation — step-dependent baseline으로 variance 감소, (4) 두 mechanism 결합으로 path quality에 정확히 target하는 gradient, (5) 3개 실세계 데이터셋에서 SOTA PRS 비교.

📊 주요 결과

3개 실세계 데이터셋에서 SOTA PRS significantly outperform.
Length-dependent bias·gradient variance 동시 감소 입증.
Path 확장이 meaningful exploration을 가능하게 함.
Code: github.com/hongruhou89/ProRL.

💭 의의 및 한계

**의의**: PRS RL의 gradient estimation 결함을 이론적으로 정밀 분석·rectify, 두 mechanism 모두 light-weight하면서 효과적, recommendation 외 sequential decision task에 일반화 가능. **한계**: 3개 데이터셋의 일반화는 추가 검증, expected reward 추정의 정확도 의존, hyperparameter 의존성은 abstract에서 미명시.

🚀 실용적 활용

E-commerce·미디어의 proactive 추천 시스템.
사용자 선호 shift 캠페인 설계.
Sequential decision RL의 gradient estimation 일반 개선.