Skip to content

deeplabcut.core.metrics.matching

Algorithms to match predictions to ground truth labels.

Classes:

Name Description
PotentialMatch

A potential match between predicted pose and ground truth pose.

Functions:

Name Description
match_greedy_oks

Greedy matching of ground truth individuals to predicted individuals using OKS.
match_greedy_rmse

Greedy matching of ground truth individuals to predicted individuals using RMSE.

PotentialMatch dataclass

A potential match between predicted pose and ground truth pose.

Parameters:

Name Type Description Default

pose

ndarray

An array of shape (num_bodyparts, 3)

 required

score

float

The score for the prediction. This could be the mean of the confidence score for each bodypart, or another value representing how confident the model is that this assembly is correct.

 required

gt

ndarray | None

None if no ground truth pose was matched to the prediction. If defined, the ground truth to which the prediction is matched. It should be of shape (num_bodyparts, 3), where the 3 values are x, y and visibility.

 None

oks

float

The OKS score between the pose and the ground truth.

 0.0

Methods:

Name Description
keypoint_scores

Returns: The confidence score for each bodypart in the predicted pose.
match

Adds a ground truth match to this PotentialMatch.
pixel_errors

Returns:
Source code in deeplabcut/core/metrics/matching.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
@dataclass
class PotentialMatch:
    """A potential match between predicted pose and ground truth pose.

    Args:
        pose: An array of shape (num_bodyparts, 3)
        score: The score for the prediction. This could be the mean of the confidence
            score for each bodypart, or another value representing how confident the
            model is that this assembly is correct.
        gt: None if no ground truth pose was matched to the prediction. If defined, the
            ground truth to which the prediction is matched. It should be of shape
            (num_bodyparts, 3), where the 3 values are x, y and visibility.
        oks: The OKS score between the pose and the ground truth.
    """

    pose: np.ndarray
    score: float
    gt: np.ndarray | None = None
    oks: float = 0.0

    def keypoint_scores(self) -> np.ndarray:
        """Returns: The confidence score for each bodypart in the predicted pose."""
        return self.pose[:, 2].copy()

    def pixel_errors(self) -> np.ndarray:
        """
        Returns:
            The distance (in pixels) between each predicted and ground truth bodypart.
            If this prediction is unmatched, returns an array of length num_bodyparts
            containing all NaNs.
        """
        if self.gt is None:
            return np.full(len(self.pose), np.nan)

        return np.linalg.norm(self.pose[:, :2] - self.gt[:, :2], axis=1)

    def match(self, gt: np.ndarray, oks: float) -> None:
        """Adds a ground truth match to this PotentialMatch.

        Args:
            gt: The ground truth to which the prediction is matched. The ground truth
                pose should be of shape (num_bodyparts, 3), where the 3 values are x, y
                and visibility.
            oks: The OKS similarity between the ground truth and this.
        """
        self.gt = gt
        self.oks = oks

    @classmethod
    def from_pose(cls, pose: np.ndarray) -> PotentialMatch:
        assert len(pose.shape) == 2  # Must be pose for a single individual
        scores = pose[:, 2]
        if np.all(np.isnan(scores)):
            raise ValueError(f"Cannot create a Match from a pose prediction where all scores are nan (pose={pose})")

        return PotentialMatch(pose=pose, score=np.nanmean(scores).item())

keypoint_scores 

keypoint_scores() -> np.ndarray

Returns: The confidence score for each bodypart in the predicted pose.

Source code in deeplabcut/core/metrics/matching.py 
40
41
42
def keypoint_scores(self) -> np.ndarray:
    """Returns: The confidence score for each bodypart in the predicted pose."""
    return self.pose[:, 2].copy()

match 

match(gt: ndarray, oks: float) -> None

Adds a ground truth match to this PotentialMatch.

Parameters:

Name Type Description Default

gt

ndarray

The ground truth to which the prediction is matched. The ground truth pose should be of shape (num_bodyparts, 3), where the 3 values are x, y and visibility.

 required

oks

float

The OKS similarity between the ground truth and this.

 required
Source code in deeplabcut/core/metrics/matching.py 
56
57
58
59
60
61
62
63
64
65
66
def match(self, gt: np.ndarray, oks: float) -> None:
    """Adds a ground truth match to this PotentialMatch.

    Args:
        gt: The ground truth to which the prediction is matched. The ground truth
            pose should be of shape (num_bodyparts, 3), where the 3 values are x, y
            and visibility.
        oks: The OKS similarity between the ground truth and this.
    """
    self.gt = gt
    self.oks = oks

pixel_errors 

pixel_errors() -> np.ndarray

Returns:

Type Description
ndarray

The distance (in pixels) between each predicted and ground truth bodypart. If this prediction is unmatched, returns an array of length num_bodyparts containing all NaNs.
Source code in deeplabcut/core/metrics/matching.py 
44
45
46
47
48
49
50
51
52
53
54
def pixel_errors(self) -> np.ndarray:
    """
    Returns:
        The distance (in pixels) between each predicted and ground truth bodypart.
        If this prediction is unmatched, returns an array of length num_bodyparts
        containing all NaNs.
    """
    if self.gt is None:
        return np.full(len(self.pose), np.nan)

    return np.linalg.norm(self.pose[:, :2] - self.gt[:, :2], axis=1)

match_greedy_oks 

match_greedy_oks(
    ground_truth: ndarray, predictions: ndarray, oks_matrix: ndarray, oks_threshold: float = 0.0
) -> list[PotentialMatch]

Greedy matching of ground truth individuals to predicted individuals using OKS.

This is done in the same way as done in pycocotools. The predictions must be sorted by score before being passed to this function.

Parameters:

Name Type Description Default

ground_truth

ndarray

The ground truth labels for an image, of shape (n_idv, n_bpt, 2)

 required

predictions

ndarray

The predictions for an image, of shape (n_idv, n_bpt, 2)

 required

oks_matrix

ndarray

A matrix of shape (n_pred, n_kpts) where entry (i, j) is the OKS between prediction i and ground truth j.

 required

oks_threshold

float

The min. OKS for a prediction to be matched to a GT pose

 0.0

Returns:

Type Description
list[PotentialMatch]

A list containing a PotentialMatch for each predicted pose in the given predictions.
Source code in deeplabcut/core/metrics/matching.py 
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
def match_greedy_oks(
    ground_truth: np.ndarray,
    predictions: np.ndarray,
    oks_matrix: np.ndarray,
    oks_threshold: float = 0.0,
) -> list[PotentialMatch]:
    """Greedy matching of ground truth individuals to predicted individuals using OKS.

    This is done in the same way as done in pycocotools. The predictions must be sorted
    by score before being passed to this function.

    Args:
        ground_truth: The ground truth labels for an image, of shape (n_idv, n_bpt, 2)
        predictions: The predictions for an image, of shape (n_idv, n_bpt, 2)
        oks_matrix: A matrix of shape (n_pred, n_kpts) where entry (i, j) is the OKS
            between prediction i and ground truth j.
        oks_threshold: The min. OKS for a prediction to be matched to a GT pose

    Returns:
        A list containing a PotentialMatch for each predicted pose in the given
        predictions.
    """
    matches = [PotentialMatch.from_pose(pose=pred) for pred in predictions]
    matched_gt_indices = set()
    for idx, _pred in enumerate(predictions):
        oks = oks_matrix[idx]
        if np.all(np.isnan(oks)):
            continue

        ind_best = np.nanargmax(oks)

        # if this gt already matched, continue
        if ind_best in matched_gt_indices:
            continue

        # Only match the pred to the GT if the OKS value is above a given threshold
        if oks[ind_best] < oks_threshold:
            continue

        matched_gt_indices.add(ind_best)
        matches[idx].match(gt=ground_truth[ind_best], oks=oks[ind_best])

    return matches

match_greedy_rmse 

match_greedy_rmse(ground_truth: ndarray, predictions: ndarray, keep_assemblies: bool = True) -> list[PotentialMatch]

Greedy matching of ground truth individuals to predicted individuals using RMSE.

The predictions must be sorted by score before being passed to this function.

Parameters:

Name Type Description Default

ground_truth

ndarray

The ground truth labels for an image, of shape (n_idv, n_bpt, 2)

 required

predictions

ndarray

The predictions for an image, of shape (n_idv, n_bpt, 2)

 required

keep_assemblies

bool

Whether to match predicted keypoints to ground truth keypoints while enforcing that all bodyparts for a predicted individual are matched to bodyparts from the same ground truth assembly. When set to False, this corresponds to detection RMSE score.

 True

Returns:

Type Description
list[PotentialMatch]

A list containing a PotentialMatch for each predicted pose in the given predictions.
Source code in deeplabcut/core/metrics/matching.py 
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
def match_greedy_rmse(
    ground_truth: np.ndarray,
    predictions: np.ndarray,
    keep_assemblies: bool = True,
) -> list[PotentialMatch]:
    """Greedy matching of ground truth individuals to predicted individuals using RMSE.

    The predictions must be sorted by score before being passed to this function.

    Args:
        ground_truth: The ground truth labels for an image, of shape (n_idv, n_bpt, 2)
        predictions: The predictions for an image, of shape (n_idv, n_bpt, 2)
        keep_assemblies: Whether to match predicted keypoints to ground truth keypoints
            while enforcing that all bodyparts for a predicted individual are matched
            to bodyparts from the same ground truth assembly. When set to False, this
            corresponds to detection RMSE score.

    Returns:
        A list containing a PotentialMatch for each predicted pose in the given
        predictions.
    """
    if not keep_assemblies:
        raise NotImplementedError()

    matches = [PotentialMatch.from_pose(pose=pred) for pred in predictions]
    matched_gt_indices = set()
    for idx, pred in enumerate(predictions):
        bpt_distances = np.linalg.norm(pred[:, :2] - ground_truth[:, :, :2], axis=-1)
        if np.all(np.isnan(bpt_distances)):
            continue

        distances = np.nanmean(bpt_distances, axis=-1)
        ind_best = np.nanargmin(distances)

        # if this gt already matched, continue
        if ind_best in matched_gt_indices:
            continue

        matched_gt_indices.add(ind_best)
        matches[idx].match(
            gt=ground_truth[ind_best],
            oks=float("nan"),  # don't compute OKS here
        )

    return matches