cursus.processing.datasets.pipeline_datasets

class PipelineDataset(*args, **kwargs)

Bases: Dataset

Custom dataset for multimodal input supporting text, tabular, categorical, and Parquet/CSV/TSV file formats with per-column processing pipelines.

load_data(**kwargs)

Load data from file (CSV, TSV, or Parquet).

load_dataframe(dataframe)

Load data directly from a provided DataFrame.

fill_missing_value(**kwargs)

Ensure all fields are numeric or categorical with default fill values.

add_pipeline(field_name, processor_pipeline)

Adds a processing pipeline for a specified field. The pipeline is built by composing Processors via the >> operator. For example, for the text field ‘dialogue’, you might have:

pipeline = (HTMLNormalizerProcessor() >> EmojiRemoverProcessor() >>

TextNormalizationProcessor() >> DialogueSplitterProcessor() >> DialogueChunkerProcessor(tokenizer, max_tokens=512) >> TokenizationProcessor(tokenizer, add_special_tokens=True))

which you then add via:

dataset.add_pipeline(“dialogue”, pipeline)

set_text_field_name(text_name)

set_label_field_name(label_name)

set_cat_field_list(cat_field_list)

set_full_field_list(full_field_list)