ReinforcementTuningAutoraterScorer

Autorater config for evaluation.

The prompt for an autorater to scorer the parsed sample response. This field supports the following placeholders that will be replaced before scoring: - {{prompt}} - {{response}} - {{system_instruction}} - {{references.key}}

Parses autorater returned response for scoring. For example, if the autorater response has reward stored in the 2.0 block, defining a parsing response config using regex ".*(.*?)" will return a score "2.0".

Scores autorater responses by using string match reward scorer.

Scores autorater responses by directly converting parsed autorater response to a float reward. Note: Reward is clipped to be within [-1, 1], i.e., reward = max(min(reward, 1.0), -1.0).