Skip to content

deeplabcut.pose_estimation_pytorch.data.preprocessor

Helpers to run preprocess data before running inference.

Classes:

Name Description
AugmentImage

Adds an offset and scale key to the context:
ComposePreprocessor

Class to preprocess an image and turn it into a batch of inputs before running
ComputeBoundingBoxesFromCondKeypoints

Generates bounding boxes from predicted keypoints.
FilterInvalidBoundingBoxes

Filters out poses and bounding boxes that are invalid (e.g., area too small).
FilterLowConfidencePoses

Filters out poses with low confidence scores.
LoadImage

Loads an image from a file, if not yet loaded.
Preprocessor

Class to preprocess an image and turn it into a batch of inputs before running
ToBatch

Adds a batch dimension to the image tensor.
ToTensor

Transforms lists and numpy arrays into tensors.
TopDownCrop

Crops bounding boxes out of images for top-down pose estimation.

Functions:

Name Description
build_bottom_up_preprocessor

Creates a preprocessor for bottom-up pose estimation (or object detection)
build_conditional_top_down_preprocessor

Creates a preprocessor for conditional top-down pose estimation.
build_top_down_preprocessor

Creates a preprocessor for top-down pose estimation.

AugmentImage

Bases: Preprocessor

Adds an offset and scale key to the context

offset: (x, y) position of the pixel in the top left corner of the augmented image in the original image scale: size of the original image divided by the size of the new image

This allows to map the position of predictions in the transformed image back to the original image space. p_original = p_transformed * scale + offset p_transformed = (p_original - offset) / scale

Methods:

Name Description
update_offsets_and_scales

X = x' * scale' + offset'.
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
class AugmentImage(Preprocessor):
    """

    Adds an offset and scale key to the context:
        offset: (x, y) position of the pixel in the top left corner of the augmented
            image in the original image
        scale: size of the original image divided by the size of the new image

    This allows to map the position of predictions in the transformed image back to the
    original image space.
        p_original = p_transformed * scale + offset
        p_transformed = (p_original - offset) / scale
    """

    def __init__(self, transform: A.BaseCompose) -> None:
        self.transform = transform

    @staticmethod
    def get_offsets_and_scales(
        h: int,
        w: int,
        output_bboxes: list[tuple[float, float, float, float]],
    ) -> tuple[list[tuple[float, float]], list[tuple[float, float]]]:
        offsets, scales = [], []
        for bbox in output_bboxes:
            x_origin, y_origin, w_out, h_out = bbox
            x_scale, y_scale = w / w_out, h / h_out
            x_offset = -x_origin * x_scale
            y_offset = -y_origin * y_scale
            offsets.append((x_offset, y_offset))
            scales.append((x_scale, y_scale))

        return offsets, scales

    @staticmethod
    def update_offset(
        offset: tuple[float, float],
        scale: tuple[float, float],
        new_offset: tuple[float, float],
    ) -> tuple[float, float]:
        return (
            scale[0] * new_offset[0] + offset[0],
            scale[1] * new_offset[1] + offset[1],
        )

    @staticmethod
    def update_scale(scale: tuple[float, float], new_scale: tuple[float, float]) -> tuple[float, float]:
        return scale[0] * new_scale[0], scale[1] * new_scale[1]

    @staticmethod
    def update_offsets_and_scales(context, new_offsets, new_scales) -> tuple:
        """X = x' * scale' + offset'.

        x' = x'' * scale'' + offset''
        -> x = x'' * (scale' * scale'') + (scale' * offset'' + offset')
        """
        # scales and offsets are either both lists or both tuples
        offsets = context.get("offsets", (0, 0))
        scales = context.get("scales", (1, 1))
        if isinstance(offsets, tuple):
            if isinstance(new_offsets, list):
                updated_offsets = [
                    AugmentImage.update_offset(offsets, scales, new_offset) for new_offset in new_offsets
                ]
                updated_scales = [AugmentImage.update_scale(scales, new_scale) for new_scale in new_scales]
            else:
                if not len(offsets) == len(new_offsets):
                    raise ValueError("Cannot rescale lists when not same length")

                updated_offsets = AugmentImage.update_offset(offsets, scales, new_offsets)
                updated_scales = AugmentImage.update_scale(scales, new_scales)
        else:
            if isinstance(new_offsets, list):
                if not len(offsets) == len(new_offsets):
                    raise ValueError("Cannot rescale lists when not same length")

                updated_offsets = [
                    AugmentImage.update_offset(offset, scale, new_offset)
                    for offset, scale, new_offset in zip(offsets, scales, new_offsets, strict=False)
                ]
                updated_scales = [
                    AugmentImage.update_scale(scale, new_scale)
                    for scale, new_scale in zip(scales, new_scales, strict=False)
                ]
            else:
                updated_offsets = [
                    AugmentImage.update_offset(offset, scale, new_offsets)
                    for offset, scale in zip(offsets, scales, strict=False)
                ]
                updated_scales = [AugmentImage.update_scale(scale, new_scales) for scale in scales]
        return updated_offsets, updated_scales

    def __call__(self, image: Image, context: Context) -> tuple[np.ndarray, Context]:
        # If the image is a batch, process each entry
        if len(image.shape) == 4:
            batch_size, h, w, _ = image.shape
            if batch_size == 0:
                # no images in top-down when no detections
                offsets, scales = (0, 0), (1, 1)
            else:
                transformed = [
                    self.transform(
                        image=img,
                        keypoints=[],
                        class_labels=[],
                        bboxes=[[0, 0, w, h]],
                        bbox_labels=["image"],
                    )
                    for img in image
                ]
                image = np.stack([t["image"] for t in transformed])
                output_bboxes = [t["bboxes"][0] for t in transformed]
                offsets, scales = self.get_offsets_and_scales(h, w, output_bboxes)
        else:
            h, w, _ = image.shape
            transformed = self.transform(
                image=image,
                keypoints=[],
                class_labels=[],
                bboxes=[[0, 0, w, h]],
                bbox_labels=["image"],
            )
            image = transformed["image"]
            output_bboxes = [transformed["bboxes"][0]]
            offsets, scales = self.get_offsets_and_scales(h, w, output_bboxes)
            offsets = offsets[0]
            scales = scales[0]

        offsets, scales = self.update_offsets_and_scales(context, offsets, scales)
        context["offsets"] = offsets
        context["scales"] = scales
        return image, context

update_offsets_and_scales staticmethod 

update_offsets_and_scales(context, new_offsets, new_scales) -> tuple

X = x' * scale' + offset'.

x' = x'' * scale'' + offset'' -> x = x'' * (scale' * scale'') + (scale' * offset'' + offset')

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
@staticmethod
def update_offsets_and_scales(context, new_offsets, new_scales) -> tuple:
    """X = x' * scale' + offset'.

    x' = x'' * scale'' + offset''
    -> x = x'' * (scale' * scale'') + (scale' * offset'' + offset')
    """
    # scales and offsets are either both lists or both tuples
    offsets = context.get("offsets", (0, 0))
    scales = context.get("scales", (1, 1))
    if isinstance(offsets, tuple):
        if isinstance(new_offsets, list):
            updated_offsets = [
                AugmentImage.update_offset(offsets, scales, new_offset) for new_offset in new_offsets
            ]
            updated_scales = [AugmentImage.update_scale(scales, new_scale) for new_scale in new_scales]
        else:
            if not len(offsets) == len(new_offsets):
                raise ValueError("Cannot rescale lists when not same length")

            updated_offsets = AugmentImage.update_offset(offsets, scales, new_offsets)
            updated_scales = AugmentImage.update_scale(scales, new_scales)
    else:
        if isinstance(new_offsets, list):
            if not len(offsets) == len(new_offsets):
                raise ValueError("Cannot rescale lists when not same length")

            updated_offsets = [
                AugmentImage.update_offset(offset, scale, new_offset)
                for offset, scale, new_offset in zip(offsets, scales, new_offsets, strict=False)
            ]
            updated_scales = [
                AugmentImage.update_scale(scale, new_scale)
                for scale, new_scale in zip(scales, new_scales, strict=False)
            ]
        else:
            updated_offsets = [
                AugmentImage.update_offset(offset, scale, new_offsets)
                for offset, scale in zip(offsets, scales, strict=False)
            ]
            updated_scales = [AugmentImage.update_scale(scale, new_scales) for scale in scales]
    return updated_offsets, updated_scales

ComposePreprocessor

Bases: Preprocessor

Class to preprocess an image and turn it into a batch of inputs before running inference.

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
164
165
166
167
168
169
170
171
172
173
174
class ComposePreprocessor(Preprocessor):
    """Class to preprocess an image and turn it into a batch of inputs before running
    inference."""

    def __init__(self, components: list[Preprocessor]) -> None:
        self.components = components

    def __call__(self, image: Image, context: Context) -> tuple[Image, Context]:
        for preprocessor in self.components:
            image, context = preprocessor(image, context)
        return image, context

ComputeBoundingBoxesFromCondKeypoints

Bases: Preprocessor

Generates bounding boxes from predicted keypoints.

Parameters:

Name Type Description Default

cond_kpt_key

str

The key under which cond. keypoints are stored in the context.

 'cond_kpts'

bbox_margin

int

The margin to add around keypoints when generating bounding boxes.

 0

Methods:

Name Description
__call__

TODO: numpy implementation
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
class ComputeBoundingBoxesFromCondKeypoints(Preprocessor):
    """Generates bounding boxes from predicted keypoints.

    Args:
        cond_kpt_key: The key under which cond. keypoints are stored in the context.
        bbox_margin: The margin to add around keypoints when generating bounding boxes.
    """

    def __init__(self, cond_kpt_key: str = "cond_kpts", bbox_margin: int = 0) -> None:
        self.cond_kpt_key = cond_kpt_key
        self.bbox_margin = bbox_margin

    def __call__(self, image: np.ndarray, context: Context) -> tuple[np.ndarray, Context]:
        """TODO: numpy implementation"""
        if "cond_kpts" not in context:
            raise ValueError(f"Must include cond kpts to ComputeBBoxes, found {context}")

        h, w = image.shape[:2]
        context["bboxes"] = [
            bbox_from_keypoints(cond_kpts, h, w, self.bbox_margin) for cond_kpts in context[self.cond_kpt_key]
        ]
        return image, context

__call__ 

__call__(image: ndarray, context: Context) -> tuple[np.ndarray, Context]

TODO: numpy implementation

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
474
475
476
477
478
479
480
481
482
483
def __call__(self, image: np.ndarray, context: Context) -> tuple[np.ndarray, Context]:
    """TODO: numpy implementation"""
    if "cond_kpts" not in context:
        raise ValueError(f"Must include cond kpts to ComputeBBoxes, found {context}")

    h, w = image.shape[:2]
    context["bboxes"] = [
        bbox_from_keypoints(cond_kpts, h, w, self.bbox_margin) for cond_kpts in context[self.cond_kpt_key]
    ]
    return image, context

FilterInvalidBoundingBoxes

Bases: Preprocessor

Filters out poses and bounding boxes that are invalid (e.g., area too small).

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
class FilterInvalidBoundingBoxes(Preprocessor):
    """Filters out poses and bounding boxes that are invalid (e.g., area too small)."""

    def __init__(self, min_area: int = 1) -> None:
        self.min_area = min_area

    def __call__(self, image: np.ndarray, context: Context) -> tuple[np.ndarray, Context]:
        bboxes = context.get("bboxes", [])
        keypoints = context.get("cond_kpts", [])

        valid_bboxes = []
        valid_indices = []

        for i, bbox in enumerate(bboxes):
            _, _, w, h = bbox
            if w * h >= self.min_area:
                valid_bboxes.append(bbox)
                valid_indices.append(i)

        context["bboxes"] = valid_bboxes
        context["cond_kpts"] = keypoints[valid_indices]

        return image, context

FilterLowConfidencePoses

Bases: Preprocessor

Filters out poses with low confidence scores.

By default, the confidence associated to the pose is the max confidence value.

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
class FilterLowConfidencePoses(Preprocessor):
    """Filters out poses with low confidence scores.

    By default, the confidence associated to the pose is the max confidence value.
    """

    def __init__(
        self,
        confidence_threshold: float = 0.05,
        aggregate_func: Callable[[np.ndarray], float] = lambda arr: np.nanmax(arr, axis=1),
    ) -> None:
        self.confidence_threshold = confidence_threshold
        self.aggregate_func = aggregate_func

    def __call__(self, image: np.ndarray, context: Context) -> tuple[np.ndarray, Context]:
        if "cond_kpts" not in context:
            raise ValueError(f"Must include cond_kpts, found {context}")

        keypoints = context["cond_kpts"]

        if 0 in keypoints.shape:
            # No poses to filter; return early
            return image, context

        confidences = keypoints[:, :, 2]
        aggregated_confidence = self.aggregate_func(confidences)
        mask = aggregated_confidence >= self.confidence_threshold
        context["cond_kpts"] = keypoints[mask]

        return image, context

LoadImage

Bases: Preprocessor

Loads an image from a file, if not yet loaded.

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
177
178
179
180
181
182
183
184
185
186
187
188
189
class LoadImage(Preprocessor):
    """Loads an image from a file, if not yet loaded."""

    def __init__(self, color_mode: str = "RGB") -> None:
        self.color_mode = color_mode

    def __call__(self, image: Image, context: Context) -> tuple[np.ndarray, Context]:
        if isinstance(image, (str, Path)):
            image = load_image(image, color_mode=self.color_mode)

        h, w = image.shape[:2]
        context["image_size"] = w, h
        return image, context

Preprocessor

Bases: ABC

Class to preprocess an image and turn it into a batch of inputs before running inference.

As an example, a pre-processor can load an image, use a "bboxes" key from context to crop bounding boxes for individuals (going from a (h, w, 3) array to a (num_individuals, h, w, 3) array), and convert it into a tensor ready for inference.

Methods:

Name Description
__call__

Pre-processes an image.
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
class Preprocessor(ABC):
    """Class to preprocess an image and turn it into a batch of inputs before running
    inference.

    As an example, a pre-processor can load an image, use a "bboxes" key from context to
    crop bounding boxes for individuals (going from a (h, w, 3) array to a
    (num_individuals, h, w, 3) array), and convert it into a tensor ready for inference.
    """

    @abstractmethod
    def __call__(self, image: Image, context: Context) -> tuple[Image, Context]:
        """Pre-processes an image.

        Args:
            image: an image (containing height, width and channel dimensions) or a
                batch of images linked to a single input (containing an extra batch
                dimension)
            context: the context for this image or batch of images (such as bounding
                boxes, conditional pose, ...)

        Returns:
            the pre-processed image (or batch of images) and their context
        """
        pass

__call__ abstractmethod 

__call__(image: Image, context: Context) -> tuple[Image, Context]

Pre-processes an image.

Parameters:

Name Type Description Default

image

Image

an image (containing height, width and channel dimensions) or a batch of images linked to a single input (containing an extra batch dimension)

 required

context

Context

the context for this image or batch of images (such as bounding boxes, conditional pose, ...)

 required

Returns:

Type Description
tuple[Image, Context]

the pre-processed image (or batch of images) and their context
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
@abstractmethod
def __call__(self, image: Image, context: Context) -> tuple[Image, Context]:
    """Pre-processes an image.

    Args:
        image: an image (containing height, width and channel dimensions) or a
            batch of images linked to a single input (containing an extra batch
            dimension)
        context: the context for this image or batch of images (such as bounding
            boxes, conditional pose, ...)

    Returns:
        the pre-processed image (or batch of images) and their context
    """
    pass

ToBatch

Bases: Preprocessor

Adds a batch dimension to the image tensor.

This preprocessor is used to convert a single image tensor into a batched format by unsqueezing along the 0th dimension. This is typically required before passing the image to models that expect batched input (i.e., shape [B, C, H, W]).

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
338
339
340
341
342
343
344
345
346
347
class ToBatch(Preprocessor):
    """Adds a batch dimension to the image tensor.

    This preprocessor is used to convert a single image tensor into a batched format by
    unsqueezing along the 0th dimension. This is typically required before passing the
    image to models that expect batched input (i.e., shape `[B, C, H, W]`).
    """

    def __call__(self, image: Image, context: Context) -> tuple[np.ndarray, Context]:
        return image.unsqueeze(0), context

ToTensor

Bases: Preprocessor

Transforms lists and numpy arrays into tensors.

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
326
327
328
329
330
331
332
333
334
335
class ToTensor(Preprocessor):
    """Transforms lists and numpy arrays into tensors."""

    def __call__(self, image: Image, context: Context) -> tuple[np.ndarray, Context]:
        image = torch.tensor(image, dtype=torch.float)
        if len(image.shape) == 4:
            image = image.permute(0, 3, 1, 2)
        else:
            image = image.permute(2, 0, 1)
        return image, context

TopDownCrop

Bases: Preprocessor

Crops bounding boxes out of images for top-down pose estimation.

Parameters:

Name Type Description Default

output_size

int | tuple[int, int]

The (width, height) of crops to output

 required

margin

int

The margin to add around detected bounding boxes before cropping

 0

with_context

bool

Whether to keep context in the top-down crop

 True

Methods:

Name Description
__call__

TODO: numpy implementation
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
class TopDownCrop(Preprocessor):
    """Crops bounding boxes out of images for top-down pose estimation.

    Args:
        output_size: The (width, height) of crops to output
        margin: The margin to add around detected bounding boxes before cropping
        with_context: Whether to keep context in the top-down crop
    """

    def __init__(
        self,
        output_size: int | tuple[int, int],
        margin: int = 0,
        with_context: bool = True,
    ) -> None:
        if isinstance(output_size, int):
            output_size = (output_size, output_size)

        self.output_size = output_size
        self.margin = margin
        self.with_context = with_context

    def __call__(self, image: np.ndarray, context: Context) -> tuple[np.ndarray, Context]:
        """TODO: numpy implementation"""
        if "bboxes" not in context:
            raise ValueError(f"Must include bboxes to CropDetections, found {context}")

        images, offsets, scales = [], [], []
        for bbox in context["bboxes"]:
            crop, offset, scale = top_down_crop(
                image,
                bbox,
                self.output_size,
                margin=self.margin,
                crop_with_context=self.with_context,
            )
            images.append(crop)
            offsets.append(offset)
            scales.append(scale)

        context["offsets"] = np.array(offsets)
        context["scales"] = np.array(scales)

        # can have no bounding boxes if detector made no detections
        if len(images) == 0:
            h, w = self.output_size[1], self.output_size[0]  # output_size = (w, h)
            c = image.shape[2] if image.ndim == 3 else 1
            images = np.zeros((0, h, w, c), dtype=image.dtype)
        else:
            images = np.stack(images, axis=0)

        context["top_down_crop_size"] = self.output_size
        return images, context

__call__ 

__call__(image: ndarray, context: Context) -> tuple[np.ndarray, Context]

TODO: numpy implementation

Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
def __call__(self, image: np.ndarray, context: Context) -> tuple[np.ndarray, Context]:
    """TODO: numpy implementation"""
    if "bboxes" not in context:
        raise ValueError(f"Must include bboxes to CropDetections, found {context}")

    images, offsets, scales = [], [], []
    for bbox in context["bboxes"]:
        crop, offset, scale = top_down_crop(
            image,
            bbox,
            self.output_size,
            margin=self.margin,
            crop_with_context=self.with_context,
        )
        images.append(crop)
        offsets.append(offset)
        scales.append(scale)

    context["offsets"] = np.array(offsets)
    context["scales"] = np.array(scales)

    # can have no bounding boxes if detector made no detections
    if len(images) == 0:
        h, w = self.output_size[1], self.output_size[0]  # output_size = (w, h)
        c = image.shape[2] if image.ndim == 3 else 1
        images = np.zeros((0, h, w, c), dtype=image.dtype)
    else:
        images = np.stack(images, axis=0)

    context["top_down_crop_size"] = self.output_size
    return images, context

build_bottom_up_preprocessor 

build_bottom_up_preprocessor(color_mode: str, transform: BaseCompose) -> Preprocessor

Creates a preprocessor for bottom-up pose estimation (or object detection)

Creates a preprocessor that loads an image, runs some transform on it (such as normalization), creates a tensor from the numpy array (going from (h, w, 3) to (3, h, w)) and adds a batch dimension (so the final tensor shape is (1, 3, h, w))

Parameters:

Name Type Description Default

color_mode

str

whether to load the image as an RGB or BGR

 required

transform

BaseCompose

the transform to apply to the image

 required

Returns:

Type Description
Preprocessor

A default bottom-up Preprocessor
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def build_bottom_up_preprocessor(color_mode: str, transform: A.BaseCompose) -> Preprocessor:
    """Creates a preprocessor for bottom-up pose estimation (or object detection)

    Creates a preprocessor that loads an image, runs some transform on it (such as
    normalization), creates a tensor from the numpy array (going from (h, w, 3) to
    (3, h, w)) and adds a batch dimension (so the final tensor shape is (1, 3, h, w))

    Args:
        color_mode: whether to load the image as an RGB or BGR
        transform: the transform to apply to the image

    Returns:
        A default bottom-up Preprocessor
    """
    return ComposePreprocessor(
        components=[
            LoadImage(color_mode),
            AugmentImage(transform),
            ToTensor(),
            ToBatch(),
        ]
    )

build_conditional_top_down_preprocessor 

build_conditional_top_down_preprocessor(
    color_mode: str,
    transform: BaseCompose,
    bbox_margin: int,
    top_down_crop_size: tuple[int, int],
    top_down_crop_margin: int = 0,
    top_down_crop_with_context: bool = False,
) -> Preprocessor

Creates a preprocessor for conditional top-down pose estimation.

Creates a preprocessor that loads an image, computes bounding boxes from conditional keypoints (given as a context (through a "cond_kpts" key), crops all bounding boxes, runs some transforms on each cropped image (such as normalization), creates a tensor from the numpy array (going from (num_ind, h, w, 3) to (num_ind, 3, h, w)).

Parameters:

Name Type Description Default

color_mode

str

whether to load the image as an RGB or BGR

 required

transform

BaseCompose

the transform to apply to the image

 required

bbox_margin

int

The margin to add around keypoints when generating bounding boxes from conditional keypoints.

 required

top_down_crop_size

tuple[int, int]

the (width, height) to resize cropped bboxes to

 required

top_down_crop_margin

int

the margin to add around detected bboxes for the crop

 0

top_down_crop_with_context

bool

whether to keep context when applying the top-down crop

 False

Returns:

Type Description
Preprocessor

A default conditional top-down Preprocessor
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
def build_conditional_top_down_preprocessor(
    color_mode: str,
    transform: A.BaseCompose,
    bbox_margin: int,
    top_down_crop_size: tuple[int, int],
    top_down_crop_margin: int = 0,
    top_down_crop_with_context: bool = False,
) -> Preprocessor:
    """Creates a preprocessor for conditional top-down pose estimation.

    Creates a preprocessor that loads an image, computes bounding boxes from conditional
    keypoints (given as a context (through a "cond_kpts" key), crops all bounding boxes,
    runs some transforms on each cropped image (such as normalization), creates a tensor
    from the numpy array (going from (num_ind, h, w, 3) to (num_ind, 3, h, w)).

    Args:
        color_mode: whether to load the image as an RGB or BGR
        transform: the transform to apply to the image
        bbox_margin: The margin to add around keypoints when generating bounding boxes
            from conditional keypoints.
        top_down_crop_size: the (width, height) to resize cropped bboxes to
        top_down_crop_margin: the margin to add around detected bboxes for the crop
        top_down_crop_with_context: whether to keep context when applying the top-down crop

    Returns:
        A default conditional top-down Preprocessor
    """
    return ComposePreprocessor(
        components=[
            LoadImage(color_mode),
            FilterLowConfidencePoses(),
            ComputeBoundingBoxesFromCondKeypoints(bbox_margin=bbox_margin),
            FilterInvalidBoundingBoxes(),
            TopDownCrop(
                output_size=top_down_crop_size,
                margin=top_down_crop_margin,
                with_context=top_down_crop_with_context,
            ),
            AugmentImage(transform),
            ConditionalKeypointsToModelInputs(),
            ToTensor(),
        ]
    )

build_top_down_preprocessor 

build_top_down_preprocessor(
    color_mode: str,
    transform: BaseCompose,
    top_down_crop_size: tuple[int, int],
    top_down_crop_margin: int = 0,
    top_down_crop_with_context: bool = True,
) -> Preprocessor

Creates a preprocessor for top-down pose estimation.

Creates a preprocessor that loads an image, crops all bounding boxes given as a context (through a "bboxes" key), runs some transforms on each cropped image (such as normalization), creates a tensor from the numpy array (going from (num_ind, h, w, 3) to (num_ind, 3, h, w)).

Parameters:

Name Type Description Default

color_mode

str

whether to load the image as an RGB or BGR

 required

transform

BaseCompose

the transform to apply to the image

 required

top_down_crop_size

tuple[int, int]

the (width, height) to resize cropped bboxes to

 required

top_down_crop_margin

int

the margin to add around detected bboxes for the crop

 0

top_down_crop_with_context

bool

whether to keep context when applying the top-down crop

 True

Returns:

Type Description
Preprocessor

A default top-down Preprocessor
Source code in deeplabcut/pose_estimation_pytorch/data/preprocessor.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
def build_top_down_preprocessor(
    color_mode: str,
    transform: A.BaseCompose,
    top_down_crop_size: tuple[int, int],
    top_down_crop_margin: int = 0,
    top_down_crop_with_context: bool = True,
) -> Preprocessor:
    """Creates a preprocessor for top-down pose estimation.

    Creates a preprocessor that loads an image, crops all bounding boxes given as a
    context (through a "bboxes" key), runs some transforms on each cropped image (such
    as normalization), creates a tensor from the numpy array (going from
    (num_ind, h, w, 3) to (num_ind, 3, h, w)).

    Args:
        color_mode: whether to load the image as an RGB or BGR
        transform: the transform to apply to the image
        top_down_crop_size: the (width, height) to resize cropped bboxes to
        top_down_crop_margin: the margin to add around detected bboxes for the crop
        top_down_crop_with_context: whether to keep context when applying the top-down crop

    Returns:
        A default top-down Preprocessor
    """
    return ComposePreprocessor(
        components=[
            LoadImage(color_mode),
            TopDownCrop(
                output_size=top_down_crop_size,
                margin=top_down_crop_margin,
                with_context=top_down_crop_with_context,
            ),
            AugmentImage(transform),
            ToTensor(),
        ]
    )