evaluation.metrics
ExactMatch
class ExactMatch()A metric checking whether the predicted and desired output match. This doesn’t care what the inputs are.
Methods
__init__
def __init__()calculate
def calculate(
predicted:
actual:
)Calculate the exact match metric.
Parameters
predicted
The predicted output from the pipeline.
actual
The desired output.
Returns
1 if the predicted and actual outputs match exactly, 0 otherwise.
description
def description()UncasedMatch
class UncasedMatch()A metric for testing whether the predicted and desired outputs are matching strings. Case-insensitive and strips whitespace.
Methods
__init__
def __init__()calculate
def calculate(
predicted: str
actual: str
)Calculate the exact match metric, if the input value has been wrapped in a list
Parameters
predicted
The predicted output from the pipeline
actual: list
A list where an item is the desired output of the pipeline
Returns
1 if the predicted and actual outputs match exactly, 0 otherwise
description
def description()FuzzyMatchRatio
class FuzzyMatchRatio()A metric that compares predicted strings to desired output.
Scores are normalised InDel distance
Methods
__init__
def __init__()calculate
def calculate(
predicted: str
actual: str
)Calculates the Fuzzy Match Ratio metric
Parameters
predicted: str
String output from a SingleResultPipeline
actual: str
Ground truth, the string the pipeline is trying to predict
description
def description()calc_precision
def calc_precision(
relevant_instances: Any[List]
prediction: Any[List]
)Compares two lists and calculates precision
Parameters
relevant_instances: List[Any]
The set of relevant instances, or positive class
prediction: List[Any]
A prediction made by an information retrieval system
Returns
float
A score for the precision
calc_recall
def calc_recall(
relevant_instances: List[Any]
prediction: List[Any]
)Compares two lists and calculates recall
Parameters
relevant_instances: List[Any]
The set of relevant instances, or positive class
prediction: List[Any]
A prediction made by an information retrieval system
Returns
float
A score for the recall
PrecisionMetric
class PrecisionMetric()Methods
__init__
def __init__()calculate
def calculate(
predicted: List[Any]
actual: List[Any]
)Calculates precision for the information retrieval pipeline’s prediction against a positive set
Parameters
predicted: List[Any]
The output of an information retrieval pipeline
actual: List[Any]
The set of relevant instances for the input
description
def description()RecallMetric
class RecallMetric()Methods
__init__
def __init__()calculate
def calculate(
predicted: List[Any]
actual: List[Any]
)Calculates recall for the information retrieval pipeline’s prediction against a positive set
Parameters
predicted: List[Any]
The output of an information retrieval pipeline
actual: List[Any]
The set of relevant instances for the input
description
def description()FScoreMetric
class FScoreMetric(
beta: float
)Methods
__init__
def __init__(
beta: float
)Initialises the F-Score metric
Parameters
beta: float
The ratio by which to weight precision to recall
calculate
def calculate(
predicted: Any[List]
actual: Any[List]
)Calculates F score with the class beta for the information retrieval pipeline’s prediction against a positive set
Parameters
predicted: List[Any]
The output of an information retrieval pipeline
actual: List[Any]
The set of relevant instances for the input
description
def description()