Customize Evaluation

Janggu facilitates automatic evaluation and scoring using the Scorer callbacks for model.predict and model.evaluate.

A number of export methods are readily available in the package. In this section, we describe how to define custom scoring and export functionality to serve specialized use cases. If you intend to implement a custom scorer or a custom exporter, some of the unit test might also serve as useful examples / starting points.

Score callable

The scoring function should be a python callable with the following signature:

def custom_score(ytrue, ypred):
   """Custom score to be used with model.evaluate"""
   # do some evaluation
   return score

def custom_score(ypred):
   """Custom score to be used with model.predict"""
   # do some evaluation
   return score

If additional parameters are required for the evaluation, you might want to use the following construct

class CustomScore(object):
    def __init__(self, extra_parameter):
        self.extra_parameter

    def __call__(self, ytrue, ypred):
        # do some evaluation using self.extra_parameter
        return score

The results returned by the custom scorer may be of variable types, e.g. list or a scalar value, depending on the use case. Therefore, it is important to choose or designe an exporter that can understand and process the score subsequently.

Exporter callable

A custom exporter can be defined as a python callable using the following construct

class CustomExport(object):
    def __init__(self, extra_parameter):
        self.extra_parameter

    def __call__(self, output_dir, name, results):
        # run export
        pass

Of course, if no extra parameters are required, a plain function may also be specified to export the results.

Upon invocation of the exporter, output_dir, name and results are passed. The first two arguments dictate the output location and file name to store the results in. On the other hand, results holds the scoring results as a python dictionary of the form: {'date': <currenttime>, 'value': score_values, 'tags': datatags} score_value denotes another dictionary whose keys are given by a tuple (modelname, layername, conditionname) and whose values are the returned score values from the scoring function (see above). Example exporters can be found in API or in the source code of the package.

Custom example scorers

A Scorer maintains a name, a scoring function and an exporter function. The latter two dictate the scoring method and how the results should be stored.

An example of using Scorer to evaluate the ROC curve and the area under the ROC curve (auROC) and export it as plot and into a tsv file, respectively, is shown below

from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
from janggu import Scorer
from janggu.utils import ExportTsv
from janggu.utils import ExportScorePlot

# create a scorer
score_auroc = Scorer('auROC',
                     roc_auc_score,
                     exporter=ExportTsv())
score_roc = Scorer('ROC',
                   roc_curve,
                   exporter=ExportScorePlot(xlabel='FPR', ylabel='TPR'))
# determine the auROC
model.evaluate(DNA, LABELS, callbacks=[score_auroc, score_roc])

After the evaluation, you will find auROC.tsv and ROC.png in <results-root>/evaluation/<modelname>/.

Similarly, you can use Scorer to export the predictions of the model. Below, the output predictions are exported in json format.

from janggu import Scorer
from janggu import ExportJson

# create scorer
pred_scorer = Scorer('predict', exporter=ExportJson())

# Evaluate predictions
model.predict(DNA, callbacks=[pred_scorer])

Using the Scorer callback objects, a number of evaluations can be run out of the box. For example, with different sklearn.metrics and different exporter options. A list of available exporters can be found in API.

Alternatively, you can also plug in custom functions

# computes the per-data point loss
score_loss = Scorer('loss', lambda t, p: -t * numpy.log(p),
                         exporter=ExportJson())