add information about which model is actually used

[maurerle]

Feature request

it would be good to add some configuration information about the actually used model.
So not the config config.output_relevant_configurations which includes e.g.:

model: openai/gpt-5.5
fallback_models: ['openai/gpt-5.4', 'openai/openai-gpt-oss-120b']

but it would be good to know which of these 3 models did the review

Motivation

currently, one does not know if the fallback model was used.

[victorljii]

@maurerle
Just to clarify the current behavior: when a review runs, retry_with_fallback_models tries the primary model and then any configured fallbacks in order. The model being attempted is logged (e.g. Generating prediction with <model>), and failures are also logged (Failed to generate prediction with <model>). So from the server logs you can tell which model actually succeeded.
Would that be enough for your use case, or do you specifically need the actually used model to be shown in the PR comment / output_relevant_configurations output as well?

pr-agent/pr_agent/algo/pr_processing.py

Lines 320 to 338 in b1328df

	async def retry_with_fallback_models(f: Callable, model_type: ModelType = ModelType.REGULAR):
	all_models = _get_all_models(model_type)
	all_deployments = _get_all_deployments(all_models)
	# try each (model, deployment_id) pair until one is successful, otherwise raise exception
	for i, (model, deployment_id) in enumerate(zip(all_models, all_deployments)):
	try:
	get_logger().debug(
	f"Generating prediction with {model}"
	f"{(' from deployment ' + deployment_id) if deployment_id else ''}"
	)
	get_settings().set("openai.deployment_id", deployment_id)
	return await f(model)
	except Exception as e:
	get_logger().warning(
	f"Failed to generate prediction with {model}",
	artifact={"error": e},
	)
	if i == len(all_models) - 1: # If it's the last iteration
	raise Exception(f"Failed to generate prediction with any model of {all_models}") from e

[YokoAC]

It could be useful to add a small metadata section to the generated PR comments (configurable, e.g. disabled by default). Besides the actually used model, this could also include information such as:

• Model used (including whether a fallback model was selected?)
• Token consumption (input/output/total, if available)
• Runtime
• ...

This would make it much easier to get some information without having to inspect the server logs. In multi-user environments, not everyone has access to the docker logs or knows how to get there. The same would apply to the failure logging; seeing a certain information message that the pragent failed would be helpful for people who cannot directly check why no comments were made to the PR.