Gitlab CD crashes during deployment to google app engine - node.js

Question

I have nodeJs app that deployed on google app engine.
When I trigger deployment through GitLab CD everything goes ok.
Recently CD pipeline started returning errors, I add --verbosity=debug flag and get the following report (from GitLab CI)
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 2 files to Google Cloud Storage ═╣
╚INFO: Uploading [/builds/***/***/api/package.json] to [staging.***.appspot.com/***]
INFO: Uploading [/tmp/tmp2sf554dn/source-context.json] to [staging.***.appspot.com/***]
════════════════════════════════════════════════════════════╝
File upload done.
DEBUG: Converted YAML to JSON: "{
"entrypoint": {
"shell": ""
},
"runtime": "nodejs12"
}"
DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
Updating service [api]...
......DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
................................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
.................................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
...............................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
................................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
...............................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
............................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
.............................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
.............................DEBUG: Operation [apps/***/operations/***] not complete. Waiting to retry.
............................DEBUG: Operation [apps/***/operations/***] complete. Result: {
"error": {
"message": "Cloud build *** status: FAILURE\nError type: OK\nFull build logs: https://console.cloud.google.com/cloud-build/builds/***?project=***",
"code": 9
},
"metadata": {
"target": "***",
"method": "google.appengine.v1.Versions.CreateVersion",
"user": "***",
"endTime": "2020-09-10T09:46:28.638Z",
"#type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
"insertTime": "2020-09-10T09:45:34.795Z"
},
"done": true,
"name": "apps/MY_PROJECT/operations/***"
}
failed.
DEBUG: (gcloud.app.deploy) Error Response: [9] Cloud build *** status: FAILURE
Error type: OK
Full build logs: https://console.cloud.google.com/cloud-build/builds/***?project=***
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 983, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 808, in Run
resources = command_instance.Run(args)
File "/usr/lib/google-cloud-sdk/lib/surface/app/deploy.py", line 121, in Run
default_strategy=flex_image_build_option_default))
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 644, in RunDeploy
ignore_file=args.ignore_file)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 430, in Deploy
extra_config_settings)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 208, in DeployService
poller=done_poller)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 314, in WaitForOperation
sleep_ms=retry_interval)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 264, in WaitFor
sleep_ms, _StatusUpdate)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 326, in PollUntilDone
sleep_ms=sleep_ms)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 229, in RetryOnResult
if not should_retry(result, state):
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
return not poller.IsDone(operation)
File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 183, in IsDone
encoding.MessageToPyValue(operation.error)))
googlecloudsdk.api_lib.app.operations_util.OperationError: Error Response: [9] Cloud build *** status: FAILURE
Error type: OK
Full build logs: https://console.cloud.google.com/cloud-build/builds/***?project=***
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build *** status: FAILURE
Error type: OK
Full build logs: https://console.cloud.google.com/cloud-build/builds/***
I checked the report on cloud build and get the following:
......
Setting CNB_DEPRECATION_MODE=quiet
Setting ENTRYPOINT: '/cnb/lifecycle/launcher'
*** Images (sha256:***):
eu.gcr.io/***/app-engine-tmp/app/ttl-2h/api/buildpack-app:latest - GET https://storage.googleapis.com/eu.artifacts.***.appspot.com/containers/images/sha256:***?access_token=REDACTED: unsupported status code 404; body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: eu.artifacts.***.appspot.com/containers/images/sha256:***</Details></Error>
*** Digest: sha256:***
ERROR: failed to export: failed to write image to the following tags: [eu.gcr.io/***/app-engine-tmp/app/ttl-2h/api/buildpack-app:latest: GET https://storage.googleapis.com/eu.artifacts.***.appspot.com/containers/images/sha256:***?access_token=REDACTED: unsupported status code 404; body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: eu.artifacts.***.appspot.com/containers/images/sha256:***</Details></Error>]
I don't alter the service account which used for deployment to google app engine before
Interesting detail that when I trigger deployment locally everything goes OK, and from custom GitLab.domain, but when I start to trigger on gitlab.com crashes recently.
Maybe you have a similar experience from fixing such problems?
Thanks.