sagemaker pytorch "Artifact upload failed" - pytorch

Question

I'm running a test sagemaker pytorch training.
It creates the estimator runs the training successfully.
However it dies while runnning the "Uploading generated training model"
The error is "Error for Training job pytorch-training-2022-12-05-19-45-41-370: Failed. Reason: ClientError: Artifact upload failed:Too many files are written"
estimator = PyTorch( # create the estimator
entry_point="CloudSeg.py",
input_mode="FastFile",
TrainingInputMode='FastFile',
role=role,
py_version="py38",
framework_version="1.11.0",
instance_count=1,
instance_type="ml.g4dn.xlarge",
checkpoint_s3_uri=checkpoint_s3_bucket,
checkpoint_local_path=checkpoint_local_path,
use_spot_instances=use_spot_instances,
max_run=max_run,
max_wait=max_wait,
hyperparameters={"epochs": 1, "backend": "nccl"},
)
estimator.fit({"training": "s3://bucket/DATA/"}) # fit with the training data
The result of the fit is:
2022-12-05 19:54:10 Training - Training image download completed. Training in progress.
2022-12-05 19:54:10 Uploading - Uploading generated training model
2022-12-05 19:54:10 Failed - Training job failed
ProfilerReport-1670269542: Stopping
-
UnexpectedStatusException
Traceback (most recent call last)
/tmp/ipykernel_19821/1489485288.py in \<cell line: 1\>()
\----\> 1 estimator.fit({"training": 's3://picard-prov/38-cloud-simple-unet_DATA/'})
...
\~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
3891
3892 if wait:
\-\> 3893 self.\_check_job_status(job_name, description, "TrainingJobStatus")
3894 if dot:
3895 print()
\~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/sagemaker/session.py in \_check_job_status(self, job, desc, status_key_name)
3429 actual_status=status,
3430 )
\-\> 3431 raise exceptions.UnexpectedStatusException(
3432 message=message,
3433 allowed_statuses=\["Completed", "Stopped"\],
UnexpectedStatusException: Error for Training job pytorch-training-2022-12-05-19-45-41-370: **Failed. Reason: ClientError: Artifact upload failed:Too many files are written**
Any idea on how to solve this?
Thank you !
I tried getting rid of fastfile mode. It didn't help

Answer 1

When training completes SageMaker will process training outputs, among things, it will upload the files that CloudSeg.py placed in /opt/ml/model. Check out how many files you end up placing in these output folders that SageMaker will upload to S3 in your behalf (According to the the error message it's too many).
/opt/ml/model
/opt/ml/output
You could write code to print out the files stored in there as the last step of your algorithm, or use something SageMaker SSH Helper to interactively inspect what is happening.

Answer 2

I think I fixed the problem. I changed my "checkpoint_s3_bucket" to the session default bucket. Haven't gotten an error since.
bucket=sagemaker.Session().default_bucket()
base_job_name="sagemaker-checkpoint-test"
checkpoint_in_bucket="checkpoints"
# The S3 URI to store the checkpoints
checkpoint_s3_bucket="s3://{}/{}/{}".format(bucket, base_job_name, checkpoint_in_bucket)