cursus.steps.scripts.tabular_lookup_model_building

TabularLookupModelBuilding Processing Script.

Reads a tabular dataset (parquet) from an upstream data-loading step and turns the dataset itself into a non-parametric model, then packages it into model.tar.gz for downstream MIMS Package + Payload:

  • MODEL_KIND=lookup -> group VALUE_COLUMNS by KEY_COLUMNS into a

    {key: [values]} map, sharded into JSON.

  • MODEL_KIND=set_membership -> de-duplicated set of KEY_COLUMNS, sharded

    into JSON.

There is no training and there are no learned parameters (“the dataset IS the model”). This script does NOT write an inference handler — the per-project handler is bundled by the Package step (inference_scripts_input).

Contract (matches tabular_lookup_model_building.step.yaml):

Input: /opt/ml/processing/input/data (parquet shards) Output: /opt/ml/processing/output/model/model.tar.gz

model.tar.gz structure:
code/

config.json # the model manifest (kind, keys, values, shards) lookup/<shard>.json # {“records”: [{“key”: […], “values”: […]}]} (lookup) keyset/<shard>.json # {“keys”: [[…], …]} (set_membership)

hyperparameters.json # required by downstream MIMS packaging

Dependencies (pandas / numpy / pyarrow) are declared in the interface’s framework_requirements and provided by the SKLearn framework container — this script does no runtime pip bootstrap.

load_data(input_dir)[source]

Load parquet shards from the upstream (Cradle) output directory.

build_lookup(df, key_columns, value_columns, dedup)[source]

Group value_columns by key_columns into a list of {key, values} records.

build_keyset(df, key_columns, dedup)[source]

De-duplicated set of key_columns tuples.

build_model_directory(model_dir, model_kind, key_columns, value_columns, records, shard_count, inference_fields=None)[source]

Write the model manifest + sharded JSON into the model directory.

inference_fields is the model’s INFERENCE input schema — a list of [field_name, field_type] pairs (type NUMERIC | TEXT) describing the request keys the serving handler reads (e.g. [["normalizedAddress","TEXT"], ["saddr","TEXT"]]). It is emitted into hyperparameters.json as the tab_field_list / cat_field_list / full_field_list that the downstream Payload step (payload.py) reads to generate MIMS load-testing samples. These are the INFERENCE fields, NOT the build key_columns/value_columns (which are the source dataset’s columns and generally differ).

create_model_tarball(model_dir, output_path)[source]

Package the model directory into model.tar.gz (script tars its own artifact).

main(input_paths, output_paths, environ_vars, job_args)[source]

Entry point.

Parameters:

  • input_paths (Dict[str, str]) – {“input_data”: “/opt/ml/processing/input/data”}

  • output_paths (Dict[str, str]) – {“model_output”: “/opt/ml/processing/output/model”}

  • environ_vars (Dict[str, str]) – MODEL_KIND / KEY_COLUMNS / VALUE_COLUMNS / DEDUP / SHARD_COUNT / INFERENCE_FIELDS

  • job_args (Namespace) – parsed CLI args (unused; kept for the standard signature)