deeplabcut.utils.conversioncode

Functions:

Name	Description
adapt_labeled_data_to_new_project	Given the config.yaml file, this function will convert the labels of an ancient
analyze_videos_converth5_to_csv	By default the output poses (when running analyze_videos) are stored as
analyze_videos_converth5_to_nwb	Convert all h5 output data files in video_folder to NWB format.
convertcsv2h5	Convert annotation files in labeled-data from csv to h5.
merge_windowsannotationdataONlinuxsystem	If a project was created on Windows (and labeled there,) but ran on unix then the

adapt_labeled_data_to_new_project

adapt_labeled_data_to_new_project(
    config_path: str | Path, remove_old_bodyparts=False, other_scorer=False, userfeedback=False
)

Given the config.yaml file, this function will convert the labels of an ancient project to a new project. For this, the labeled data must be in the project folder, under the labeled-data folder and with the same configuration as all deeplabcut projects.

Parameters:

Name	Type	Description	Default
config_path	str	The path to the config.yaml file.	required
remove_old_bodyparts	bool	If True, old bodyparts not in the new project are removed from the dataframe. Defaults to False.	False
other_scorer	bool	If True, the labels will be converted to the new scorer. Defaults to False.	False
userfeedback	bool	If true the user will be asked specifically for each folder in labeled-data if the containing csv shall be converted to hdf format. Defaults to True.	False

Source code in deeplabcut/utils/conversioncode.py

def adapt_labeled_data_to_new_project(
    config_path: str | Path,
    remove_old_bodyparts=False,
    other_scorer=False,
    userfeedback=False,
):
    """Given the config.yaml file, this function will convert the labels of an ancient
    project to a new project. For this, the labeled data must be in the project folder,
    under the labeled-data folder and with the same configuration as all deeplabcut
    projects.

    Args:
        config_path (str): The path to the config.yaml file.
        remove_old_bodyparts (bool): If True, old bodyparts not in the new project are
            removed from the dataframe. Defaults to False.
        other_scorer (bool): If True, the labels will be converted to the new scorer. Defaults to False.
        userfeedback (bool): If true the user will be asked specifically
            for each folder in labeled-data if the containing csv
            shall be converted to hdf format. Defaults to True.
    """
    # Load the config file
    cfg = dlc.auxiliaryfunctions.read_config(config_path)

    # Get the Project path
    project_path = cfg["project_path"]

    # Get the bodyparts
    bodyparts = cfg["multianimalbodyparts"]
    print("New Bodyparts:", bodyparts)

    # Iterate over each labeled data video

    # Use tqdm for a progress bar
    for video in tqdm.tqdm(cfg["video_sets"]):
        print("Video:", video)

        video_name = video.split("\\")[-1]
        # discard the file extension
        video_name = video_name.split(".")[0]
        # Load the csv file
        label_path = Path(project_path) / "labeled-data" / video_name
        csv_files = [f.name for f in label_path.iterdir() if f.name.endswith(".csv")]
        if not csv_files:
            print("No csv file in the folder:", label_path)
        else:
            csv_path = str(label_path / csv_files[0])
            df = pd.read_csv(csv_path, header=None)

            # get the scorer
            if other_scorer:
                scorer = cfg["scorer"]
                # Change the scorer in the dataframe
                df.iloc[0, 3:] = pd.Series([scorer] * len(df.columns[3:]))

            else:
                scorer = df.iloc[0, 3]

            # Get the individuals
            individuals = np.unique(df.iloc[1, 3:])

            # Get the old bodyparts
            old_bodyparts = np.unique(df.iloc[2, 3:])
            print("Old bodyparts:", old_bodyparts)

            # Get the unmber of old bodyparts
            num_of_old_bodyparts = len(old_bodyparts)

            # Bodyparts to add
            print("Bodyparts to add:", set(bodyparts) - set(old_bodyparts))

            # If a bodypart is missing, add it to the dataframe
            for index, bodypart in enumerate(bodyparts):
                if bodypart not in old_bodyparts:
                    num_of_old_bodyparts += 1
                    for i, individual in enumerate(individuals):
                        # create the columns for the bodypart, concatenate, the individual, the bodypart, and nan values
                        x_column = pd.concat(
                            [
                                pd.Series(scorer),
                                pd.Series(individual),
                                pd.Series(bodypart),
                                pd.Series("x"),
                                pd.Series(np.nan, index=df.index),
                            ],
                            axis=0,
                            ignore_index=True,
                        )
                        y_column = pd.concat(
                            [
                                pd.Series(scorer),
                                pd.Series(individual),
                                pd.Series(bodypart),
                                pd.Series("y"),
                                pd.Series(np.nan, index=df.index),
                            ],
                            axis=0,
                            ignore_index=True,
                        )
                        # Insert the columns in the dataframe
                        df.insert(
                            i * 2 * num_of_old_bodyparts + index * 2 + 3,
                            "insert_" + bodypart + "_x" + individual,
                            x_column,
                        )
                        df.insert(
                            i * 2 * num_of_old_bodyparts + index * 2 + 4,
                            "insert" + bodypart + "_y" + individual,
                            y_column,
                        )

            # If the old bodyparts are not in the new project, remove them
            if remove_old_bodyparts:
                for bodypart in old_bodyparts:
                    if bodypart not in bodyparts:
                        df = df.drop(df.columns[df.iloc[2, :] == bodypart], axis=1)

            # Save the dataframe
            df.to_csv(csv_path, index=False, header=False)

    # Create/Update the h5 file
    convertcsv2h5(config_path, userfeedback=userfeedback)

analyze_videos_converth5_to_csv

analyze_videos_converth5_to_csv(video_folder, videotype='.mp4', listofvideos=False)

By default the output poses (when running analyze_videos) are stored as MultiIndex Pandas Array, which contains the name of the network, body part name, (x, y) label position in pixels, and the likelihood for each frame per body part. These arrays are stored in an efficient Hierarchical Data Format (HDF) in the same directory, where the video is stored. This functions converts hdf (h5) files to the comma-separated values format (.csv), which in turn can be imported in many programs, such as MATLAB, R, Prism, etc.

Parameters:

Name	Type	Description	Default
video_folder	string	Absolute path of a folder containing videos and the corresponding h5 data files.	required
videotype	string	Only videos with this extension are screened. Defaults to .mp4.	'.mp4'

Examples:

Converts all pose-output files belonging to mp4 videos in the folder '/media/alex/experimentaldata/cheetahvideos' to csv files:

deeplabcut.analyze_videos_converth5_to_csv(
    "/media/alex/experimentaldata/cheetahvideos",
    ".mp4",
)

Source code in deeplabcut/utils/conversioncode.py

def analyze_videos_converth5_to_csv(video_folder, videotype=".mp4", listofvideos=False):
    """By default the output poses (when running analyze_videos) are stored as
    MultiIndex Pandas Array, which contains the name of the network, body part name, (x,
    y) label position in pixels, and the likelihood for each frame per body part.
    These arrays are stored in an efficient Hierarchical Data Format (HDF) in the
    same directory, where the video is stored. This functions converts hdf (h5) files to
    the comma-separated values format (.csv), which in turn can be imported in many
    programs, such as MATLAB, R, Prism, etc.

    Args:
        video_folder (string): Absolute path of a folder containing videos and the corresponding h5 data files.
        videotype (string, optional): Only videos with this extension are screened. Defaults to .mp4.

    Examples:
        Converts all pose-output files belonging to mp4 videos in the folder
        '/media/alex/experimentaldata/cheetahvideos' to csv files:

            deeplabcut.analyze_videos_converth5_to_csv(
                "/media/alex/experimentaldata/cheetahvideos",
                ".mp4",
            )
    """
    if listofvideos:  # can also be called with a list of videos (from GUI)
        videos = video_folder  # GUI gives a list of videos
        if len(videos) > 0:
            h5_files = collect_video_paths(Path(videos[0]).parent, extensions=".h5")
        else:
            h5_files = []
    else:
        h5_files = collect_video_paths(video_folder, extensions=".h5")
        videos = collect_video_paths(video_folder, extensions=videotype)

    _convert_h5_files_to("csv", None, h5_files, videos)

analyze_videos_converth5_to_nwb

analyze_videos_converth5_to_nwb(config: str | Path, video_folder: str | Path, videotype='.mp4', listofvideos=False)

Convert all h5 output data files in video_folder to NWB format.

Parameters:

Name	Type	Description	Default
config	string	Absolute path to the project YAML config file.	required
video_folder	string	Absolute path of a folder containing videos and the corresponding h5 data files.	required
videotype	string	Only videos with this extension are screened. Defaults to .mp4.	'.mp4'

Examples:

Converts all pose-output files belonging to mp4 videos in the folder '/media/alex/experimentaldata/cheetahvideos' to NWB files:

deeplabcut.analyze_videos_converth5_to_nwb(
    config,
    "/media/alex/experimentaldata/cheetahvideos",
    ".mp4",
)

Source code in deeplabcut/utils/conversioncode.py

def analyze_videos_converth5_to_nwb(
    config: str | Path,
    video_folder: str | Path,
    videotype=".mp4",
    listofvideos=False,
):
    """Convert all h5 output data files in `video_folder` to NWB format.

    Args:
        config (string): Absolute path to the project YAML config file.
        video_folder (string): Absolute path of a folder containing videos and the corresponding h5 data files.
        videotype (string, optional): Only videos with this extension are screened. Defaults to .mp4.

    Examples:
        Converts all pose-output files belonging to mp4 videos in the folder
        '/media/alex/experimentaldata/cheetahvideos' to NWB files:

            deeplabcut.analyze_videos_converth5_to_nwb(
                config,
                "/media/alex/experimentaldata/cheetahvideos",
                ".mp4",
            )
    """
    if listofvideos:  # can also be called with a list of videos (from GUI)
        videos = video_folder  # GUI gives a list of videos
        if len(videos) > 0:
            h5_files = collect_video_paths(Path(videos[0]).parent, extensions=".h5")
        else:
            h5_files = []
    else:
        h5_files = collect_video_paths(video_folder, extensions=".h5")
        videos = collect_video_paths(video_folder, extensions=videotype)

    _convert_h5_files_to("nwb", config, h5_files, videos)

convertcsv2h5

convertcsv2h5(config: str | Path, userfeedback=True, scorer=None)

Convert annotation files in labeled-data from csv to h5.

Allows the user to manually edit the csv (e.g. to correct the scorer name and then convert it into hdf format). WARNING: conversion might corrupt the data.

Parameters:

Name	Type	Description	Default
config	str | Path	Full path of the config.yaml file as a string.	required
userfeedback	bool	If true the user will be asked specifically for each folder in labeled-data if the containing csv shall be converted to hdf format.	True
scorer	string	If a string is given, then the scorer/annotator in all csv and hdf files that are changed, will be overwritten with this name.	None

Examples:

Convert csv annotation files for reaching-task project into hdf:

deeplabcut.convertcsv2h5("/analysis/project/reaching-task/config.yaml")

Convert csv annotation files for reaching-task project into hdf while changing the scorer/annotator in all annotation files to Albert:

deeplabcut.convertcsv2h5("/analysis/project/reaching-task/config.yaml", scorer="Albert")

Source code in deeplabcut/utils/conversioncode.py

def convertcsv2h5(config: str | Path, userfeedback=True, scorer=None):
    """Convert annotation files in labeled-data from csv to h5.

    Allows the user to manually edit the csv
    (e.g. to correct the scorer name and then convert it into hdf format).
    WARNING: conversion might corrupt the data.

    Args:
        config (str | Path): Full path of the config.yaml file as a string.
        userfeedback (bool, optional): If true the user will be asked specifically
            for each folder in labeled-data if the containing csv shall be converted to hdf format.
        scorer (string, optional): If a string is given, then the scorer/annotator
            in all csv and hdf files that are changed, will be overwritten with this name.

    Examples:
        Convert csv annotation files for reaching-task project into hdf:

            deeplabcut.convertcsv2h5("/analysis/project/reaching-task/config.yaml")

        Convert csv annotation files for reaching-task project into hdf while changing
        the scorer/annotator in all annotation files to Albert:

            deeplabcut.convertcsv2h5("/analysis/project/reaching-task/config.yaml", scorer="Albert")
    """
    cfg = auxiliaryfunctions.read_config(config)
    videos = cfg["video_sets"].keys()
    video_names = [Path(i).stem for i in videos]
    folders = [Path(config).parent / "labeled-data" / Path(i) for i in video_names]
    if not scorer:
        scorer = cfg["scorer"]

    for folder in folders:
        try:
            if userfeedback:
                print("Do you want to convert the csv file in folder:", folder, "?")
                askuser = input("yes/no")
            else:
                askuser = "yes"

            if askuser in ("y", "yes", "Ja", "ha", "oui"):  # multilanguage support :)
                fn = folder / ("CollectedData_" + cfg["scorer"] + ".csv")
                # Determine whether the data are single- or multi-animal without loading into memory
                # simply by checking whether 'individuals' is in the second line of the CSV.
                with fn.open() as datafile:
                    head = list(islice(datafile, 0, 5))
                if "individuals" in head[1]:
                    header = list(range(4))
                else:
                    header = list(range(3))
                if head[-1].split(",")[0] == "labeled-data":
                    index_col = [0, 1, 2]
                else:
                    index_col = 0
                data = pd.read_csv(fn, index_col=index_col, header=header)
                data.columns = data.columns.set_levels([scorer], level="scorer")
                guarantee_multiindex_rows(data)
                data.to_hdf(fn.replace(".csv", ".h5"), key="df_with_missing", mode="w")
                data.to_csv(fn)
        except FileNotFoundError:
            print("Attention:", folder, "does not appear to have labeled data!")

merge_windowsannotationdataONlinuxsystem

merge_windowsannotationdataONlinuxsystem(cfg)

If a project was created on Windows (and labeled there,) but ran on unix then the data folders corresponding in the keys in cfg['video_sets'] are not found.

This function gets them directly by looping over all folders in labeled-data

Source code in deeplabcut/utils/conversioncode.py

def merge_windowsannotationdataONlinuxsystem(cfg):
    """If a project was created on Windows (and labeled there,) but ran on unix then the
    data folders corresponding in the keys in cfg['video_sets'] are not found.

    This function gets them directly by looping over all folders in labeled-data
    """
    AnnotationData = []
    data_path = Path(cfg["project_path"], "labeled-data")
    annotationfolders = [d for d in data_path.iterdir() if d.is_dir()]
    print("The following folders were found:", annotationfolders)
    for folder in annotationfolders:
        filename = str(Path(folder) / ("CollectedData_" + cfg["scorer"] + ".h5"))
        try:
            data = pd.read_hdf(filename)
            guarantee_multiindex_rows(data)
            AnnotationData.append(data)
        except FileNotFoundError:
            print(filename, " not found (perhaps not annotated)")

    return AnnotationData