cursus.processing.text.gensim_tokenize_processor

class GensimTokenizeProcessor(keyed_vectors, max_length=None, pad_to_max_length=True, embeddings_key='embeddings', attention_mask_key='attention_mask')

Bases: Processor

Tokenization processor that maps words to FastText embeddings. Accepts a list of text chunks, splits on whitespace, looks up embeddings, and pads/truncates to max_length. Returns a dict per chunk with:

• embeddings_key: List[List[float]] of shape (L, D)

• attention_mask_key: List[int] of shape (L,)

process(input_chunks)