evaluation.eval_tests

source

LLMPipelineTest

class LLMPipelineTest(
	name: str
	pipeline: LLMPipeline
	metrics: SingleResultMetric[list]
)

This class provides a pipeline test for LLM pipelines that return a single result

Methods

__init__

def __init__(
	name: str
	pipeline: LLMPipeline
	metrics: SingleResultMetric[list]
)

Initialises the LLMPipelineTest class

Parameters

name: str Name given to the test pipeline: LLMPipeline The pipeline used to generate output metrics: list[SingleResultMetric] A list of metrics used to compare the pipeline output with the expected output

run_pipeline

def run_pipeline(
	input_data
)

Runs the provided pipeline on the input_data

Parameters

input_data The data used for input to the pipeline

Returns

str The reply from the pipeline

evaluate

def evaluate(
	input_data
	expected_output
)

Evaluates the attached pipeline’s output against the expected output using the metrics

Parameters

input_data The data used for input to the pipeline

expected_output The expected result of running the input data through the pipeline

Returns

Dict

A dictionary of results from evaluating the pipeline.

drop_pipeline

def drop_pipeline()