cursus.steps.configs.config_data_uploading_step

Data Uploading Step Configuration.

Pydantic v2 config for the DataUploading step that uploads data from S3 to Andes via the SAIS SDK. Maps to CreateDataUploadJobRequest fields.

class DataUploadingConfig(*, author, bucket, role, region, service_name, pipeline_version, model_class='xgboost', current_date=<factory>, framework_version='2.1.0', py_version='py310', source_dir=None, enable_caching=False, use_secure_pypi=False, max_runtime_seconds=172800, project_root_folder, provider_name, table_name, table_version, cradle_account, schema_sdl, job_type=None, input_s3_location=None, input_format='CSV', has_header=True, cluster_type='MEDIUM', instance_type='ml.m5.xlarge', instance_count=1, volume_size_in_gb=30, **extra_data)[source]

Bases: BasePipelineConfig

Configuration for the DataUploading step (S3 → Andes via SAIS SDK).

provider_name: str
table_name: str
table_version: int
cradle_account: str
schema_sdl: str
job_type: str | None
input_s3_location: str | None
input_format: str
has_header: bool
cluster_type: str
instance_type: str
instance_count: int
volume_size_in_gb: int
property job_config: Dict[str, Any]

Assemble the full job config dict for CreateDataUploadJobRequest.

This is what gets written to /opt/ml/processing/config/config and deserialized via CreateDataUploadJobRequest.__from_coral__(**config). Note: inputS3Url is NOT included here — it’s injected at runtime via –input-path argument by the SDK processor.

validate_cluster_type()[source]

Validate cluster_type is a known value.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'protected_namespaces': (), 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

author: str
bucket: str
role: str
region: str
service_name: str
pipeline_version: str
model_class: str
current_date: str
framework_version: str
py_version: str
source_dir: str | None
enable_caching: bool
use_secure_pypi: bool
max_runtime_seconds: int
project_root_folder: str