Skip to content

deeplabcut.utils.video_processor

Author: Hao Wu hwu01@g.harvard.edu You can find the directory for your ffmpeg bindings by: "find / | grep ffmpeg" and then setting it.

This is the helper class for video reading and saving in DeepLabCut. Updated by AM

You can set various codecs below, fourcc = cv2.VideoWriter_fourcc(*'MJPG') i.e. 'XVID'

Classes:

Name Description
VideoProcessor

Abstract base class for video reading and writing.
VideoProcessorCV

OpenCV-backed video reader and writer.

VideoProcessor

Bases: ABC

Abstract base class for video reading and writing.

Subclasses implement backend-specific video loading, metadata extraction, output video creation, frame reading, frame writing, and cleanup.

Parameters:

Name Type Description Default

fname

str

Path to the input video. If empty, no input video is opened.

 ''

sname

str

Path to the output video. If empty, no output video is created.

 ''

nframes

int

Number of frames to process. -1 means all frames.

 -1

fps

float | None

Optional FPS override.

 None

codec

str

FourCC codec string used for output videos.

 'X264'

sh

int | Literal[''] | None

Output video height. "" and None mean use the input height.

 ''

sw

int | Literal[''] | None

Output video width. "" and None mean use the input width.

 ''

Attributes:

Name Type Description
fname str

Input video path.
sname str

Output video path.
nframes int

Number of frames to process.
video_fps float | None

Video frame rate.
FPS float | None

Legacy alias for video_fps.
h int

Input video height.
w int

Input video width.
nc int

Number of channels.
i int

Number of successfully loaded frames.
vid

Backend-specific input video object.
svid

Backend-specific output video object.
sh int

Output video height.
sw int

Output video width.
Notes

height(), width(), fps(), counter(), and frame_count() are retained as methods for backwards compatibility.

Methods:

Name Description
close

Implement your own.
create_video

Implement your own.
get_info

Implement your own.
save_frame

Implement your own.
Source code in deeplabcut/utils/video_processor.py 
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
class VideoProcessor(ABC):
    """Abstract base class for video reading and writing.

    Subclasses implement backend-specific video loading, metadata extraction,
    output video creation, frame reading, frame writing, and cleanup.

    Args:
        fname (str): Path to the input video. If empty, no input video is opened.
        sname (str): Path to the output video. If empty, no output video is created.
        nframes (int): Number of frames to process. ``-1`` means all frames.
        fps (float | None): Optional FPS override.
        codec (str): FourCC codec string used for output videos.
        sh (int | Literal[""] | None): Output video height. ``""`` and ``None``
            mean use the input height.
        sw (int | Literal[""] | None): Output video width. ``""`` and ``None``
            mean use the input width.

    Attributes:
        fname (str): Input video path.
        sname (str): Output video path.
        nframes (int): Number of frames to process.
        video_fps (float | None): Video frame rate.
        FPS (float | None): Legacy alias for ``video_fps``.
        h (int): Input video height.
        w (int): Input video width.
        nc (int): Number of channels.
        i (int): Number of successfully loaded frames.
        vid: Backend-specific input video object.
        svid: Backend-specific output video object.
        sh (int): Output video height.
        sw (int): Output video width.

    Notes:
        ``height()``, ``width()``, ``fps()``, ``counter()``, and
        ``frame_count()`` are retained as methods for backwards compatibility.
    """

    def __init__(
        self,
        fname: str = "",
        sname: str = "",
        nframes: int = -1,
        fps: float | None = None,
        codec: str = "X264",
        sh: int | Literal[""] | None = "",
        sw: int | Literal[""] | None = "",
    ):
        self._fname = None
        self._sname = None
        self._fps = None
        self._nframes = None
        self._h = 0
        self._w = 0
        self.vid = None
        self.svid = None
        self.sh = 0
        self.sw = 0
        self.fname = fname
        self.sname = sname
        self.codec = codec
        self.nframes = nframes
        self.nc = 3
        self.i = 0

        try:
            if self.fname != "":
                self.vid = self.get_video()
                self.get_info()
                self.sh = 0
                self.sw = 0
            if self.sname != "":
                if sh in ("", None) and sw in (None, ""):
                    self.sh = self._h
                    self.sw = self._w
                else:
                    self.sw = sw
                    self.sh = sh
                self.svid = self.create_video()

        except Exception as ex:
            logger.exception("VideoProcessor initialization failed: %s", ex)

        if fps is not None:  # Overwrite the video's FPS
            # NOTE @C-Achard 2026-06-09 improving checks here might break old API
            # same for raising on missing FPS
            self.fps = fps

    def load_frame(self):
        frame = self._read_frame()
        if frame is not None:
            self.i += 1
        return frame

    @property
    def fname(self):
        return self._fname

    @fname.setter
    def fname(self, value):
        self._fname = "" if value in (None, "") else str(value)

    @property
    def sname(self):
        return self._sname

    @sname.setter
    def sname(self, value):
        self._sname = "" if value in (None, "") else str(value)

    @property
    def height(self):
        return self._h

    @property
    def width(self):
        return self._w

    @height.setter
    def height(self, value):
        self._h = int(value)

    @width.setter
    def width(self, value):
        self._w = int(value)

    @property
    def fps(self):
        return self._fps

    @fps.setter
    def fps(self, value):
        self._fps = None if value is None else float(value)

    @property
    def nframes(self):
        return self._nframes

    @nframes.setter
    def nframes(self, value):
        self._nframes = int(value)

    @abstractmethod
    def get_video(self):

        raise NotImplementedError("Implement your own get_video method.")

    @abstractmethod
    def get_info(self):
        """Implement your own."""

    @abstractmethod
    def create_video(self):
        """Implement your own."""

    @abstractmethod
    def _read_frame(self):
        """Implement your own."""

    @abstractmethod
    def save_frame(self, frame):
        """Implement your own."""

    @abstractmethod
    def close(self):
        """Implement your own."""

close abstractmethod 

close()

Implement your own.

Source code in deeplabcut/utils/video_processor.py 
196
197
198
@abstractmethod
def close(self):
    """Implement your own."""

create_video abstractmethod 

create_video()

Implement your own.

Source code in deeplabcut/utils/video_processor.py 
184
185
186
@abstractmethod
def create_video(self):
    """Implement your own."""

get_info abstractmethod 

get_info()

Implement your own.

Source code in deeplabcut/utils/video_processor.py 
180
181
182
@abstractmethod
def get_info(self):
    """Implement your own."""

save_frame abstractmethod 

save_frame(frame)

Implement your own.

Source code in deeplabcut/utils/video_processor.py 
192
193
194
@abstractmethod
def save_frame(self, frame):
    """Implement your own."""

VideoProcessorCV

Bases: VideoProcessor

OpenCV-backed video reader and writer.

This implementation uses cv2.VideoCapture for reading videos and cv2.VideoWriter for writing videos. Frames returned by load_frame are converted from OpenCV's native BGR channel order to RGB channel order. Frames passed to save_frame are expected to be in RGB order and are converted back to BGR before writing.

Attributes:

Name Type Description
fname str

Path to the input video. If empty, no input video is opened.
sname str

Path to the output video. If empty, no output video is created.
nframes int

Number of frames to process. If initialized as -1, it is replaced by the total number of frames reported by OpenCV.
codec str

FourCC codec string used when creating the output video.
h int

Input video height in pixels.
w int

Input video width in pixels.
nc int

Number of channels. This implementation uses 3.
i int

Number of frames successfully loaded through load_frame().
FPS float

Frames per second reported by OpenCV, or the user-provided override.
sh int

Output video height in pixels.
sw int

Output video width in pixels.
vid VideoCapture | None

OpenCV video reader.
svid VideoWriter | None

OpenCV video writer.

Methods:

Name Description
close

Release OpenCV reader and writer resources.
create_video

Create an OpenCV video writer.
get_info

Populate metadata from the OpenCV video reader.
get_video

Open the input video with OpenCV.
save_frame

Write one RGB frame to the output video.
Source code in deeplabcut/utils/video_processor.py 
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
class VideoProcessorCV(VideoProcessor):
    """OpenCV-backed video reader and writer.

    This implementation uses cv2.VideoCapture for reading videos and
    cv2.VideoWriter for writing videos. Frames returned by
    `load_frame` are converted from OpenCV's native BGR channel order to
    RGB channel order. Frames passed to `save_frame` are expected to be in
    RGB order and are converted back to BGR before writing.

    Attributes:
        fname (str): Path to the input video. If empty, no input video is opened.
        sname (str): Path to the output video. If empty, no output video is created.
        nframes (int): Number of frames to process. If initialized as ``-1``,
            it is replaced by the total number of frames reported by OpenCV.
        codec (str): FourCC codec string used when creating the output video.
        h (int): Input video height in pixels.
        w (int): Input video width in pixels.
        nc (int): Number of channels. This implementation uses ``3``.
        i (int): Number of frames successfully loaded through ``load_frame()``.
        FPS (float): Frames per second reported by OpenCV, or the user-provided
            override.
        sh (int): Output video height in pixels.
        sw (int): Output video width in pixels.
        vid (cv2.VideoCapture | None): OpenCV video reader.
        svid (cv2.VideoWriter | None): OpenCV video writer.
    """

    def get_video(self):
        """Open the input video with OpenCV.

        Returns:
            cv2.VideoCapture: OpenCV video capture object for ``self.fname``.
        """
        return cv2.VideoCapture(self.fname)

    def get_info(self):
        """Populate metadata from the OpenCV video reader.

        Sets:
            self.w: Frame width in pixels.
            self.h: Frame height in pixels.
            self.nframes: Number of frames to process.
            self.FPS: Frames per second reported by OpenCV.
            self.nc: Number of channels, always ``3``.

        Notes:
            If ``self.nframes`` is ``-1`` or greater than the total number of
            frames reported by OpenCV, it is replaced by OpenCV's frame count.
        """
        self.width = int(self.vid.get(cv2.CAP_PROP_FRAME_WIDTH))
        self.height = int(self.vid.get(cv2.CAP_PROP_FRAME_HEIGHT))
        all_frames = int(self.vid.get(cv2.CAP_PROP_FRAME_COUNT))
        self.fps = self.vid.get(cv2.CAP_PROP_FPS)
        self.nc = 3

        if self.nframes == -1 or self.nframes > all_frames:
            self.nframes = all_frames

    def create_video(self):
        """Create an OpenCV video writer.

        Returns:
            cv2.VideoWriter: OpenCV video writer for ``self.sname``.

        Notes:
            ``self.sw`` and ``self.sh`` are expected to be set by the base class
            before this method is called. The codec is interpreted as a FourCC
            string, preserving the historical OpenCV behavior.
        """
        fourcc = cv2.VideoWriter_fourcc(*self.codec)
        return cv2.VideoWriter(self.sname, fourcc, self.fps, (self.sw, self.sh), True)

    def _read_frame(self):
        """Read the next video frame.

        Returns:
            numpy.ndarray | None: The next frame in RGB channel order, or
            ``None`` if no frame could be read.

        Notes:
            OpenCV returns BGR frames. This method converts them to RGB using
            ``np.flip(frame, 2)`` to preserve legacy behavior.
        """
        if self.vid is None:
            return None

        success, frame = self.vid.read()
        if not success:
            return frame

        return np.flip(frame, 2)

    def save_frame(self, frame):
        """Write one RGB frame to the output video.

        Args:
            frame (numpy.ndarray | None): RGB frame to write. ``None`` is ignored.

        Notes:
            This method preserves the historical behavior of silently ignoring
            ``None`` frames. Non-``None`` frames are converted from RGB to BGR
            before being passed to OpenCV.
        """
        if frame is None:
            return
        if self.svid is not None:
            self.svid.write(np.flip(frame, 2))
        else:
            logger.warning(f"Could not write video because no output video writer is open for {self.sname}")

    def close(self):
        """Release OpenCV reader and writer resources.

        This method is safe to call multiple times. After release, ``self.svid``
        and ``self.vid`` are set to ``None`` to avoid accidental reuse of closed
        OpenCV handles.
        """
        if hasattr(self, "svid") and self.svid is not None:
            self.svid.release()
            self.svid = None

        if hasattr(self, "vid") and self.vid is not None:
            self.vid.release()
            self.vid = None

close 

close()

Release OpenCV reader and writer resources.

This method is safe to call multiple times. After release, self.svid and self.vid are set to None to avoid accidental reuse of closed OpenCV handles.

Source code in deeplabcut/utils/video_processor.py 
311
312
313
314
315
316
317
318
319
320
321
322
323
324
def close(self):
    """Release OpenCV reader and writer resources.

    This method is safe to call multiple times. After release, ``self.svid``
    and ``self.vid`` are set to ``None`` to avoid accidental reuse of closed
    OpenCV handles.
    """
    if hasattr(self, "svid") and self.svid is not None:
        self.svid.release()
        self.svid = None

    if hasattr(self, "vid") and self.vid is not None:
        self.vid.release()
        self.vid = None

create_video 

create_video()

Create an OpenCV video writer.

Returns:

Type Description

cv2.VideoWriter: OpenCV video writer for self.sname.
Notes

self.sw and self.sh are expected to be set by the base class before this method is called. The codec is interpreted as a FourCC string, preserving the historical OpenCV behavior.

Source code in deeplabcut/utils/video_processor.py 
259
260
261
262
263
264
265
266
267
268
269
270
271
def create_video(self):
    """Create an OpenCV video writer.

    Returns:
        cv2.VideoWriter: OpenCV video writer for ``self.sname``.

    Notes:
        ``self.sw`` and ``self.sh`` are expected to be set by the base class
        before this method is called. The codec is interpreted as a FourCC
        string, preserving the historical OpenCV behavior.
    """
    fourcc = cv2.VideoWriter_fourcc(*self.codec)
    return cv2.VideoWriter(self.sname, fourcc, self.fps, (self.sw, self.sh), True)

get_info 

get_info()

Populate metadata from the OpenCV video reader.

Sets

self.w: Frame width in pixels. self.h: Frame height in pixels. self.nframes: Number of frames to process. self.FPS: Frames per second reported by OpenCV. self.nc: Number of channels, always 3.

Notes

If self.nframes is -1 or greater than the total number of frames reported by OpenCV, it is replaced by OpenCV's frame count.

Source code in deeplabcut/utils/video_processor.py 
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
def get_info(self):
    """Populate metadata from the OpenCV video reader.

    Sets:
        self.w: Frame width in pixels.
        self.h: Frame height in pixels.
        self.nframes: Number of frames to process.
        self.FPS: Frames per second reported by OpenCV.
        self.nc: Number of channels, always ``3``.

    Notes:
        If ``self.nframes`` is ``-1`` or greater than the total number of
        frames reported by OpenCV, it is replaced by OpenCV's frame count.
    """
    self.width = int(self.vid.get(cv2.CAP_PROP_FRAME_WIDTH))
    self.height = int(self.vid.get(cv2.CAP_PROP_FRAME_HEIGHT))
    all_frames = int(self.vid.get(cv2.CAP_PROP_FRAME_COUNT))
    self.fps = self.vid.get(cv2.CAP_PROP_FPS)
    self.nc = 3

    if self.nframes == -1 or self.nframes > all_frames:
        self.nframes = all_frames

get_video 

get_video()

Open the input video with OpenCV.

Returns:

Type Description

cv2.VideoCapture: OpenCV video capture object for self.fname.
Source code in deeplabcut/utils/video_processor.py 
228
229
230
231
232
233
234
def get_video(self):
    """Open the input video with OpenCV.

    Returns:
        cv2.VideoCapture: OpenCV video capture object for ``self.fname``.
    """
    return cv2.VideoCapture(self.fname)

save_frame 

save_frame(frame)

Write one RGB frame to the output video.

Parameters:

Name Type Description Default

frame

ndarray | None

RGB frame to write. None is ignored.

 required
Notes

This method preserves the historical behavior of silently ignoring None frames. Non-None frames are converted from RGB to BGR before being passed to OpenCV.

Source code in deeplabcut/utils/video_processor.py 
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
def save_frame(self, frame):
    """Write one RGB frame to the output video.

    Args:
        frame (numpy.ndarray | None): RGB frame to write. ``None`` is ignored.

    Notes:
        This method preserves the historical behavior of silently ignoring
        ``None`` frames. Non-``None`` frames are converted from RGB to BGR
        before being passed to OpenCV.
    """
    if frame is None:
        return
    if self.svid is not None:
        self.svid.write(np.flip(frame, 2))
    else:
        logger.warning(f"Could not write video because no output video writer is open for {self.sname}")