[Proposal] Allow separate episode counts for conditions evaluated in parallel within one Run

[Papaercold]

Proposal

Hi, I'm Zihan from Lightwheel.

Allow users to configure num_episodes separately for different evaluation conditions executed in parallel within the same Run.

For example, if some parallel envs evaluate condition A and others evaluate condition B, users should be able to request 100 episodes for A and 200 for B. Each condition should have its own completion count.

Motivation

Different evaluation conditions may require different sample counts and take different amounts of time to finish. Users should be able to specify how many episodes to evaluate for each condition, independently of execution speed.

Separate Runs already support their own episode budgets. This request concerns different conditions within one parallel Run.

Alternatives

Put each condition in a separate Run and configure its episode count there. It would be useful to express the same allocation within one parallel Run as well.

Build Info

• Arena Release: main / development version, package version 0.3.0.

Additional context

A concrete existing use case is RigidObjectSet: parallel envs can run the same pick-and-place task with different object members. With ordered assignment, one env can evaluate sweet potato and another jug, with those object identities fixed across resets. The requested counts would apply to these object conditions. This does not assume support for arbitrary different task classes within one vectorized Run.

A per-env quota would cover the case where each condition is fixed to one env. If multiple envs evaluate the same condition, they should share that condition's quota.

The existing global episode-budget mode could remain available.

Checklist

• I have checked that there is no similar issue in the repo (required)

Acceptance Criteria

• Users can specify independent episode counts for different conditions within one parallel Run.
• Completed episodes count toward their own condition's quota, including unsuccessful or timed-out episodes.
• Evaluation completes when all requested quotas are met; an early stop is reported as incomplete.