Skip to content

deeplabcut.pose_estimation_pytorch.models.modules.norm

Normalization layers.

Classes:

Name Description
ScaleNorm

Implementation of ScaleNorm.

ScaleNorm

Bases: Module

Implementation of ScaleNorm.

ScaleNorm was introduced in "Transformers without Tears: Improving the Normalization of Self-Attention".

Code based on the mmpose implementation. See https://github.com/open-mmlab/mmpose for more details.

Parameters:

Name Type Description Default

dim

int

The dimension of the scale vector.

 required

eps

float

The minimum value in clamp.

 1e-05
Source code in deeplabcut/pose_estimation_pytorch/models/modules/norm.py 
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
class ScaleNorm(nn.Module):
    """Implementation of ScaleNorm.

    ScaleNorm was introduced in "Transformers without Tears: Improving the Normalization
    of Self-Attention".

    Code based on the `mmpose` implementation. See https://github.com/open-mmlab/mmpose
    for more details.

    Args:
        dim: The dimension of the scale vector.
        eps: The minimum value in clamp.
    """

    def __init__(self, dim: int, eps: float = 1e-5):
        super().__init__()
        self.scale = dim**-0.5
        self.eps = eps
        self.g = nn.Parameter(torch.ones(1))

    def forward(self, x):
        norm = torch.linalg.norm(x, dim=-1, keepdim=True)
        norm = norm * self.scale
        return x / norm.clamp(min=self.eps) * self.g