cursus.processing.dataloaders.pipeline_dataloader

build_collate_batch(input_ids_key='input_ids', attention_mask_key='attention_mask')

Build a collate function for models with text modalities.

Handles: - Single or multiple text modalities (e.g., chat, shiptrack) with tokenization keys - Tabular features - Labels

All text modalities use the same tokenizer output keys since they share the same tokenizer.

Parameters:

• input_ids_key (str) – Key name for text input_ids (applies to all text modalities)

• attention_mask_key (str) – Key name for text attention_mask (applies to all text modalities)

Returns:

Collate function for DataLoader

build_trimodal_collate_batch(input_ids_key='input_ids', attention_mask_key='attention_mask')

Build a collate function for models with text modalities.

Handles: - Single or multiple text modalities (e.g., chat, shiptrack) with tokenization keys - Tabular features - Labels

All text modalities use the same tokenizer output keys since they share the same tokenizer.

Parameters:

• input_ids_key (str) – Key name for text input_ids (applies to all text modalities)

• attention_mask_key (str) – Key name for text attention_mask (applies to all text modalities)

Returns:

Collate function for DataLoader