[pznger]

PR: Add Multi-granular Reward Weight Ablation Study

📋 Summary

This PR introduces a comprehensive ablation study framework for the multi-granular reward weights introduced in commit f70215a. It includes:

1. Training script enhancement - Add --reward_config parameter to train_grpo.py for custom reward weights
2. Ablation experiment script - Automated framework for running 9 ablation experiments
3. Experiment design documentation - Complete methodology, hypotheses, and analysis plan

🎯 Motivation

The current multi-granular reward function uses weights (format=1, tool=3, argument=2, process=1, outcome=3) based on prior design, but lacks systematic validation. This ablation study aims to answer:

• Is differentiated weighting necessary compared to uniform weights?
• How does sparse (outcome-only) vs. dense (5-component) reward perform?
• What is the relative importance of each component?
• Is the 30-step curriculum schedule beneficial?

Current best model (8B GRPO v3) achieves strict 72.8% on 81-task benchmark. This study provides empirical evidence for future optimization.

📦 Changes

1. Core Training Enhancement

File: AgenticArxiv/rl/train_grpo.py

Change: Add optional --reward_config parameter

def main(
    # ... existing parameters ...
    reward_curriculum_steps: int = 30,
    reward_config: str = None,  # NEW: path to reward config JSON
    num_generations: int = 4,
    # ... rest of parameters ...
):

Usage:

# Default behavior (unchanged)
python -m AgenticArxiv.rl.train_grpo --model outputs/sft/final

# With custom weights
python -m AgenticArxiv.rl.train_grpo --model outputs/sft/final \
    --reward_config artifacts/ablation/config.json

Backward compatibility: ✅ 100% compatible. Existing scripts work without modification.

2. Ablation Experiment Script

File: scripts/ablation_reward_weights.py

Features:

• 9 experiment configurations (baseline, uniform, outcome-only, etc.)
• Automated training + evaluation pipeline
• Result aggregation and summary
• Dry-run mode for configuration validation
• Quick-test mode (10 steps) for smoke testing

Usage:

# Run all 9 experiments (100 steps each)
python scripts/ablation_reward_weights.py --base_model outputs/sft/final

# Quick test (10 steps)
python scripts/ablation_reward_weights.py --quick_test

# Dry run (generate configs only)
python scripts/ablation_reward_weights.py --dry_run

# Run specific experiments
python scripts/ablation_reward_weights.py --experiments baseline uniform

3. Experiment Design Documentation

File: docs/ablation_reward_weights.md

Contents:

• Background and motivation
• 9 experiment groups with detailed configurations
• 5 testable hypotheses
• Analysis methodology (training stability, evaluation metrics, component-level analysis)
• Expected outcomes and optimization recommendations
• Limitations and future work

🧪 Experiment Groups

Group	Weights (format, tool, argument, process, outcome)	Curriculum	Validates
Baseline	(1, 3, 2, 1, 3)	30 steps	Current configuration
Uniform	(1, 1, 1, 1, 1)	30 steps	Need for differentiation
OutcomeOnly	(0, 0, 0, 0, 3)	30 steps	Dense vs. sparse reward
NoOutcome	(1, 3, 2, 1, 0)	30 steps	Importance of outcome
StrongFormat	(3, 3, 2, 1, 3)	30 steps	Format correctness emphasis
ToolFocus	(1, 5, 1, 1, 3)	30 steps	Tool selection accuracy
ArgumentFocus	(1, 3, 4, 1, 3)	30 steps	Parameter matching
ProcessRich	(1, 3, 2, 3, 3)	30 steps	Dense process feedback
NoCurriculum	(1, 3, 2, 1, 3)	0 steps	Curriculum necessity

📊 Expected Output

After running experiments, the following artifacts are generated:

artifacts/ablation_reward_weights/
├── experiments_meta.json          # Experiment metadata
├── ablation_summary.json          # Aggregated results
├── baseline/
│   ├── config.json                # Weight configuration
│   ├── training.log               # Training logs
│   ├── final/                     # Trained model
│   └── eval/
│       ├── summary.json           # Evaluation metrics
│       └── traces.jsonl           # Trajectory records
├── uniform/
│   └── ...
└── ...

Key metrics:

• strict_success_rate: Strict task completion
• tool_accuracy: Tool sequence accuracy
• argument_accuracy: Parameter matching accuracy
• reward_mean/std: Reward statistics
• frac_reward_zero_std: Training stability indicator

✅ Testing

Unit Test (Dry Run)

$ python scripts/ablation_reward_weights.py --dry_run
🔬 将运行 9 个实验组
📁 输出目录: artifacts/ablation_reward_weights
============================================================
实验组: baseline
描述: 当前默认配置（基线）
权重: format=1.0, tool=3.0, argument=2.0, process=1.0, outcome=3.0
课程学习步数: 30
...
🔍 Dry run 模式，跳过实际训练
✅ 配置已保存至: artifacts/ablation_reward_weights/baseline/config.json
...
✅ 所有实验完成

Integration Test (Quick Test)

$ python scripts/ablation_reward_weights.py --quick_test --experiments baseline
⚡ 快速测试模式：10 steps
🔬 将运行 1 个实验组
...
🚀 训练命令: python -m AgenticArxiv.rl.train_grpo --model outputs/sft/final \
    --reward_config artifacts/ablation_reward_weights/baseline/config.json ...
✅ 训练完成
📊 开始评测...
✅ 评测完成

🔄 Migration Guide

For existing users: No action needed. All existing commands work as before.

For ablation study users:

1. Apply the patch to train_grpo.py (see train_grpo_patch.md)
2. Run experiments: python scripts/ablation_reward_weights.py
3. Analyze results from artifacts/ablation_reward_weights/ablation_summary.json

📚 References

• Multi-granular reward design: docs/multigranular_rl.md (commit f70215a)
• Curriculum learning analysis: scripts/analyze_curriculum.py
• Current best model: 8B GRPO v3 (strict 72.8%), see docs/edge-vlm-posttrain.md
• Reward design inspired by LLM-TIR (Tool Invocation Reward)

🙏 Acknowledgements

This ablation study builds upon the multi-granular reward framework introduced in commit f70215a and references the project's existing curriculum learning and evaluation infrastructure.

---

Checklist:

• Code follows project style
• Backward compatible (no breaking changes)
• Documentation added
• Tested with dry-run mode
• Config files follow project conventions

Estimated runtime:

• Full experiments (9 groups × 100 steps): ~4-6 hours on A100
• Quick test (1 group × 10 steps): ~5 minutes