pipeline.feature_extraction package

Submodules

pipeline.feature_extraction.facemesh_features module

pipeline.feature_extraction.facemesh_features.get_facemesh_features(srv_video: str, dst_csv: str)

Extract facemesh features from a video file for with mediapipe

Parameters

  • src_video (str) – Path to src video

  • dst_csv (str) – Path to dst feature csv

Returns

face features dataframe with features x,y,z as columns with the format [feature type]_[number]

Return type

df (DataFrame)

pipeline.feature_extraction.format_openface module

pipeline.feature_extraction.format_openface.drop_columns(df: pandas.core.frame.DataFrame, drop_key: str, drop_list: list = ['face_id', 'timestamp', 'success'])

Drop columns according to key or custom drop list

Parameters

  • df (pd.DataFrame) – [description]

  • drop_key (str) – [description]

  • drop_list (list, optional) – [description]. Defaults to [“face_id”, “timestamp”, “success”].

Returns

clean df without columns

Return type

[type]

pipeline.feature_extraction.format_openface.main(args)
pipeline.feature_extraction.format_openface.parse_args(args)
pipeline.feature_extraction.format_openface.reformat_csv(src_path: pathlib.Path, dst_path: pathlib.Path) → pandas.core.frame.DataFrame

Drops unused columns and removes duplicate rows

This function is intended for cleaning up an openface csv with a single person detected.

Parameters

  • src_path (Path) – path to openface csv

  • dst_path (Path) – path to openface csv

Returns

[description]

Return type

pd.DataFrame

pipeline.feature_extraction.format_openface.remove_duplicates(df: pandas.core.frame.DataFrame) → pandas.core.frame.DataFrame

Remove duplicate rows from an Openface CSV

TODO: double check this function is correct (is partially redundant?)

Parameters

df (pd.DataFrame) – dataframe to clean

Returns

cleaned dataframe

Return type

pd.DataFrame

pipeline.feature_extraction.json_to_csv module

pipeline.feature_extraction.json_to_csv.get_keypoints(data)

Get the keypoints detected in the frame ‘data’ is the info read in by read_json(folder, frame), basically a json file loaded into useable form

pipeline.feature_extraction.json_to_csv.main(json_path, csv_path)
pipeline.feature_extraction.json_to_csv.read_json(file)

pipeline.feature_extraction.librosa_features module

pipeline.feature_extraction.multi_person_openpose module

pipeline.feature_extraction.multi_person_openpose.center_distance(c1, c2)
pipeline.feature_extraction.multi_person_openpose.create_cluster(x, y)
pipeline.feature_extraction.multi_person_openpose.get_cluster_center(cluster)
pipeline.feature_extraction.multi_person_openpose.get_keyframe_difference(keyframe1, keyframe2)

Measure the mean absolute-valued difference between non-0 values of two equal-length numpy arrays This is used to compare between non-0 keypoints of detected people across frames

pipeline.feature_extraction.multi_person_openpose.get_keypoint_center(keyframe)
pipeline.feature_extraction.multi_person_openpose.get_keypoints(data)

Get the keypoints detected in the frame ‘data’ is the info read in by read_json(folder, frame), basically a json file loaded into useable form

pipeline.feature_extraction.multi_person_openpose.get_point(keyframe, keypoint)

Given ‘data’ (extracted json data from one video frame), a specified person (int), and a keypoint (0 for nose, 1 for neck, etc) return the x and y coordinate

pipeline.feature_extraction.multi_person_openpose.main(n, input_video, json_path, output_folder)
pipeline.feature_extraction.multi_person_openpose.read_json(folder, frame)
pipeline.feature_extraction.multi_person_openpose.repeat_row(df)

add a copy of the last row to the dataframe

pipeline.feature_extraction.multi_person_openpose.sort_people(cl, df, data_curr, n)

cl: the array of clusters df: the array of dataframes data_curr: the extracted json data of the current frame

pipeline.feature_extraction.multi_person_openpose.update_cluster(cluster, keyframe_center)

pipeline.feature_extraction.openSMILE_features_depricated module

pipeline.feature_extraction.openSMILE_features_depricated.get_opensmile_features(paths: dict, feature_types: list = ['gemaps'])

High level runner for opensmile

TODO: increase flexibility of use. Fix egemaps?

Parameters

  • paths (dict) – contains paths to config, audio_file, csv_dir, and smile_exe

  • feature_types (list, optional) – [description]. Defaults to [“gemaps”].

pipeline.feature_extraction.openSMILE_features_depricated.main(args)
pipeline.feature_extraction.openSMILE_features_depricated.parse_args(args)

pipeline.feature_extraction.openface_features module

pipeline.feature_extraction.openface_features.get_openface_features(openface_exe: str, src_video: str, out_dir: str, cam_width: int = 1280, cam_height: int = 720)

Extract facial features with OpenFace

Parameters

  • openface_exe (str) – path to OpenFace executable (OpenFace/build/bin/FeatureExtraction)

  • src_video (str) – path to video file to get features from

  • out_dir (str) – path to destination to put features and other outputs

  • cam_width (int, optional) – [description]. Defaults to 1280.

  • cam_height (int, optional) – [description]. Defaults to 720.

pipeline.feature_extraction.openface_features.main(args)
pipeline.feature_extraction.openface_features.parse_args(args)

pipeline.feature_extraction.opensmile_features module

pipeline.feature_extraction.voice_activity_features module

class pipeline.feature_extraction.voice_activity_features.Frame(bytes, timestamp, duration)

Bases: object

Represents a “frame” of audio data.

pipeline.feature_extraction.voice_activity_features.frame_generator(frame_duration_ms, audio, sample_rate)

Generates audio frames from PCM audio data. Takes the desired frame duration in milliseconds, the PCM data, and the sample rate. Yields Frames of the requested duration.

pipeline.feature_extraction.voice_activity_features.get_voice_activity(src_audio: str, aggressiveness: int, dst_dir: str)

The voice activity in an audio file is detected and stored.

Parameters

  • src_audio (str) – path to src audio

  • aggressiveness (int) – an integer between 0 and 3. 0 is the least aggressive about filtering out non-speech, 3 is the most aggressive.

  • dst_dir (str) – path to dst for saving json and chunks

Output:

The voice activity that is detected is saved as a json. This is useful for later annotation, but you may prefer to convert the json to a csv.

pipeline.feature_extraction.voice_activity_features.main(args)
pipeline.feature_extraction.voice_activity_features.parse_args(args)
pipeline.feature_extraction.voice_activity_features.read_wave(path)

Reads a .wav file. Takes the path, and returns (PCM audio data, sample rate).

pipeline.feature_extraction.voice_activity_features.vad_collector(sample_rate, frame_duration_ms, padding_duration_ms, vad, frames)

Filters out non-voiced audio frames. Given a webrtcvad.Vad and a source of audio frames, yields only the voiced audio. Uses a padded, sliding window algorithm over the audio frames. When more than 90% of the frames in the window are voiced (as reported by the VAD), the collector triggers and begins yielding audio frames. Then the collector waits until 90% of the frames in the window are unvoiced to detrigger. The window is padded at the front and back to provide a small amount of silence or the beginnings/endings of speech around the voiced frames. Arguments: sample_rate - The audio sample rate, in Hz. frame_duration_ms - The frame duration in milliseconds. padding_duration_ms - The amount to pad the window, in milliseconds. vad - An instance of webrtcvad.Vad. frames - a source of audio frames (sequence or generator). Returns: A generator that yields PCM audio data.

pipeline.feature_extraction.voice_activity_features.write_wave(path, audio, sample_rate)

Writes a .wav file. Takes path, PCM audio data, and sample rate.

pipeline.feature_extraction.window_features module

class pipeline.feature_extraction.window_features.Windowing(csv, config='./config/windowing_default.yml')

Bases: object

Basic wrapper on pandas for windowing and subsampling a dataset.

The windowing class is intended to provide a convenient wrapper on pandas rolling functionality. There is currently a distinction between two types of features:

  • continous features: produce mean and variance across the window.

  • binary features: produce any of [median, mode, max]

feature types are specified in [example]_config.yml which lists the headers of columns according to the type of data contained (continuous or binary).

Returns

windowed dataframe without the original features.

window_binary(window_size, mode='mode')
window_dataframe(window_size=30, step=10, keep_original=False)
window_float(window_size)
pipeline.feature_extraction.window_features.get_args()
pipeline.feature_extraction.window_features.main()

Module contents