cursus.steps.scripts.xgboost_model_eval

class RiskTableMappingProcessor(column_name, label_name, smooth_factor=0.0, count_threshold=0, risk_tables=None)

Bases: object

A processor that performs risk-table-based mapping on a specified categorical variable. The ‘process’ method (called via __call__) handles single values. The ‘transform’ method handles pandas Series or DataFrames.

get_name()

set_risk_tables(risk_tables)

fit(data)

process(input_value)

Process a single input value (for the configured ‘column_name’), mapping it to its binned risk value. This method is called when the processor instance is called as a function.

transform(data)

Transform data using the computed risk tables. - If data is a DataFrame, transforms the ‘column_name’ Series within it. - If data is a Series, transforms the Series (assumed to be the target column). - If data is a single value, uses the ‘process’ method.

get_risk_tables()

class NumericalVariableImputationProcessor(variables=None, imputation_dict=None, strategy='mean')

Bases: object

A processor that performs imputation on numerical variables using predefined or computed values. Supports mean, median, and mode imputation strategies.

get_name()

fit(X, y=None)

process(input_data)

transform(X)

Transform input data by imputing missing values.

Parameters:

X (DataFrame | Series) – Input DataFrame or Series

Returns:

Transformed DataFrame or Series with imputed values

Return type:

DataFrame | Series

get_params()

load_dataframe_with_format(file_path)

Load DataFrame and detect its format.

Parameters:

file_path (Path) – Path to the file

Returns:

Tuple of (DataFrame, format_string)

Return type:

Tuple[DataFrame, str]

save_dataframe_with_format(df, output_path, format_str)

Save DataFrame in specified format.

Parameters:

• df (DataFrame) – DataFrame to save

• output_path (Path) – Base output path (without extension)

• format_str (str) – Format to save in (‘csv’, ‘tsv’, or ‘parquet’)

Returns:

Path to saved file

Return type:

Path

decompress_model_artifacts(model_dir)

Checks for a model.tar.gz file in the model directory and extracts it.

load_model_artifacts(model_dir)

Load the trained XGBoost model and all preprocessing artifacts from the specified directory. Returns model, risk_tables, impute_dict, feature_columns, and hyperparameters.

preprocess_eval_data(df, feature_columns, risk_tables, impute_dict)

Apply risk table mapping and numerical imputation to the evaluation DataFrame. Ensures all features are numeric and columns are ordered as required by the model. Preserves any non-feature columns like id and label.

log_metrics_summary(metrics, is_binary=True)

Log a nicely formatted summary of metrics for easy visibility in logs.

Parameters:

• metrics (Dict[str, int | float | str]) – Dictionary of metrics to log

• is_binary (bool) – Whether these are binary classification metrics

compute_metrics_binary(y_true, y_prob)

Compute binary classification metrics: AUC-ROC, average precision, and F1 score.

compute_metrics_multiclass(y_true, y_prob, n_classes)

Compute multiclass metrics: one-vs-rest AUC-ROC, average precision, F1 for each class, and micro/macro averages for all metrics.

compute_comparison_metrics(y_true, y_new_score, y_prev_score, is_binary=True)

Compute comparison metrics between new model and previous model scores.

Parameters:

• y_true (ndarray) – True labels

• y_new_score (ndarray) – New model prediction scores

• y_prev_score (ndarray) – Previous model prediction scores

• is_binary (bool) – Whether this is binary classification

Returns:

Dictionary of comparison metrics

Return type:

Dict[str, float]

perform_statistical_tests(y_true, y_new_score, y_prev_score, is_binary=True)

Perform statistical significance tests comparing model performances.

Parameters:

• y_true (ndarray) – True labels

• y_new_score (ndarray) – New model prediction scores

• y_prev_score (ndarray) – Previous model prediction scores

• is_binary (bool) – Whether this is binary classification

Returns:

Dictionary of statistical test results

Return type:

Dict[str, float]

load_eval_data(eval_data_dir)

Load the first data file found in the evaluation data directory. Returns a pandas DataFrame and the detected format.

get_id_label_columns(df, id_field, label_field)

Determine the ID and label columns in the DataFrame. Falls back to the first and second columns if not found.

save_predictions(ids, y_true, y_prob, id_col, label_col, output_eval_dir, input_format='csv')

Save predictions preserving input format, including id, true label, and class probabilities.

save_metrics(metrics, output_metrics_dir)

Save computed metrics as a JSON file.

plot_and_save_roc_curve(y_true, y_score, output_dir, prefix='')

Plot ROC curve and save as JPG.

plot_and_save_pr_curve(y_true, y_score, output_dir, prefix='')

Plot Precision-Recall curve and save as JPG.

plot_comparison_roc_curves(y_true, y_new_score, y_prev_score, output_dir)

Plot side-by-side ROC curves comparing new and previous models.

plot_comparison_pr_curves(y_true, y_new_score, y_prev_score, output_dir)

Plot side-by-side Precision-Recall curves comparing new and previous models.

plot_score_scatter(y_new_score, y_prev_score, y_true, output_dir)

Plot scatter plot of new vs previous model scores, colored by true labels.

plot_score_distributions(y_new_score, y_prev_score, y_true, output_dir)

Plot score distributions for both models, separated by true labels.

evaluate_model(model, df, feature_columns, id_col, label_col, hyperparams, output_eval_dir, output_metrics_dir, input_format='csv')

Run model prediction and evaluation, then save predictions and metrics preserving format. Also generate and save ROC and PR curves as JPG.

evaluate_model_with_comparison(model, df, feature_columns, id_col, label_col, previous_scores, hyperparams, output_eval_dir, output_metrics_dir, comparison_metrics, statistical_tests, comparison_plots, input_format='csv')

Run model prediction and evaluation with comparison to previous model scores preserving format. Generates comprehensive comparison metrics, statistical tests, and visualizations.

save_predictions_with_comparison(ids, y_true, y_prob, previous_scores, id_col, label_col, output_eval_dir, input_format='csv')

Save predictions preserving input format, including id, true label, new model probabilities, and previous model scores.

create_comparison_report(metrics, output_metrics_dir, is_binary)

Create a comprehensive comparison report summarizing model performance differences.

create_health_check_file(output_path)

Create a health check file to signal script completion.

main(input_paths, output_paths, environ_vars, job_args)

Main entry point for XGBoost model evaluation script. Loads model and data, runs evaluation, and saves results.

Parameters:

• input_paths (Dict[str, str]) – Dictionary of input paths

• output_paths (Dict[str, str]) – Dictionary of output paths

• environ_vars (Dict[str, str]) – Dictionary of environment variables

• job_args (argparse.Namespace) – Command line arguments