Evaluation batch

evaluation_batch ¶

Assemble single-agent evaluation results with complete objective metadata.

Notes

Scorer dimension dictionaries remain in metadata and are also forwarded to objective scores when present for every example. Partial dimensional batches fail before reaching Pareto tracking.

Examples:

Build a scalar-only evaluation batch:

batch = build_evaluation_batch(
    outputs=["answer"],
    scores=[0.8],
    trajectories=None,
    metadata=[{}],
    inputs=["question"],
    failed_indices=[],
)
assert batch.objective_scores is None

build_evaluation_batch ¶

build_evaluation_batch(
    *,
    outputs: list[str],
    scores: list[float],
    trajectories: list[ADKTrajectory] | None,
    metadata: list[dict[str, Any]],
    inputs: list[str],
    failed_indices: list[int],
) -> EvaluationBatch[ADKTrajectory, str]

Build an ordered batch with optional metadata and objective scores.

PARAMETER	DESCRIPTION
`outputs`	Agent outputs in input order. TYPE: `list[str]`
`scores`	Scalar scores aligned with outputs. TYPE: `list[float]`
`trajectories`	Captured trajectories when requested. TYPE: `list[ADKTrajectory] \| None`
`metadata`	Scorer metadata aligned with outputs. TYPE: `list[dict[str, Any]]`
`inputs`	Original example input strings. TYPE: `list[str]`
`failed_indices`	Indices whose evaluation failed. TYPE: `list[int]`

RETURNS	DESCRIPTION
`EvaluationBatch[ADKTrajectory, str]`	The assembled batch, omitting metadata when every entry is empty.

RAISES	DESCRIPTION
`ConfigurationError`	If only some examples provide non-empty dimensions.

Source code in src/gepa_adk/adapters/evolution/evaluation_batch.py

def build_evaluation_batch(
    *,
    outputs: list[str],
    scores: list[float],
    trajectories: list[ADKTrajectory] | None,
    metadata: list[dict[str, Any]],
    inputs: list[str],
    failed_indices: list[int],
) -> EvaluationBatch[ADKTrajectory, str]:
    """Build an ordered batch with optional metadata and objective scores.

    Args:
        outputs: Agent outputs in input order.
        scores: Scalar scores aligned with outputs.
        trajectories: Captured trajectories when requested.
        metadata: Scorer metadata aligned with outputs.
        inputs: Original example input strings.
        failed_indices: Indices whose evaluation failed.

    Returns:
        The assembled batch, omitting metadata when every entry is empty.

    Raises:
        ConfigurationError: If only some examples provide non-empty dimensions.
    """
    has_metadata = any(meta and isinstance(meta, dict) for meta in metadata)
    return EvaluationBatch(
        outputs=outputs,
        scores=scores,
        trajectories=trajectories,
        metadata=metadata if has_metadata else None,
        objective_scores=_objective_scores(metadata),
        inputs=inputs,
        failed_indices=failed_indices,
    )