Camel.datagen.evol instruct.scorer

BaseScorer

class BaseScorer(ABC):

score

def score(self, reference_prompt: str, candidate_prompt: str):

Compare a candidate prompt against a reference prompt and return a tuple of scores. The higher the score, the better. For example, (diversity, difficulty, feasibility).

MathScorer

class MathScorer(BaseScorer):

init

def __init__(self, agent: Optional[ChatAgent] = None):

score

def score(self, reference_problem: str, new_problem: str):

Evaluates the new math problem relative to the reference math problem.

Parameters:

• reference_problem (str): The reference math problem.
• new_problem (str): The new or evolved math problem.

Returns:

Dict[str, int]: A dictionary with scores for diversity, difficulty, validity, and solvability.

GeneralScorer

class GeneralScorer(BaseScorer):

init

def __init__(self, agent: Optional[ChatAgent] = None):

score

def score(self, reference_problem: str, new_problem: str):

Evaluates the new problem against the reference problem using structured scoring.

Parameters:

• reference_problem (str): The original problem.
• new_problem (str): The evolved or new problem.

Returns:

Dict[str, int]: A dictionary with scores for diversity, complexity, and validity.