Recently I have needed to add web sockets to my backend application currently hosted on Google App Engine (GAE) standard environment. Because web sockets are a feature only available in GAE's flexible environment, I have been attempting a redeployment but with little success.
To make the change to a flexible environment I have updated the app.yaml file from
runtime: nodejs10
env: standard
to
runtime: nodejs
env: flex
While previously working in the standard environment, now with env: flex when I run the command gcloud app deploy --app-yaml=app-staging.yaml --verbosity=debug I get the following stack trace:
Do you want to continue (Y/n)? Y
DEBUG: No bucket specified, retrieving default bucket.
DEBUG: Using bucket [gs://staging.finnsalud.appspot.com].
DEBUG: Service [appengineflex.googleapis.com] is already enabled for project [finnsalud]
Beginning deployment of service [finnsalud-staging]...
INFO: Using ignore file at [~/checkouts/twilio/backend/.gcloudignore].
DEBUG: not expecting type '<class 'NoneType'>'
Traceback (most recent call last):
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 982, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
resources = command_instance.Run(args)
File "/google-cloud-sdk/lib/surface/app/deploy.py", line 115, in Run
return deploy_util.RunDeploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 669, in RunDeploy
deployer.Deploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 428, in Deploy
source_files = source_files_util.GetSourceFiles(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/source_files_util.py", line 184, in GetSourceFiles
return list(it)
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/gcloudignore.py", line 233, in GetIncludedFiles
six.ensure_str(upload_directory), followlinks=True):
File "//google-cloud-sdk/lib/third_party/six/__init__.py", line 884, in ensure_str
raise TypeError("not expecting type '%s'" % type(s))
TypeError: not expecting type '<class 'NoneType'>'
ERROR: gcloud crashed (TypeError): not expecting type '<class 'NoneType'>'
In this stack trace, it mentions an error in google-cloud-sdk/lib/googlecloudsdk/command_lib/util/gcloudignore.py so I had also reviewed my .gcloudignore file but was unable to find anything out of place:
.gcloudignore
.git
.gitignore
node_modules/
In an attempt to work around this bug I tried removing my .gcloudignore file which resulted in a different error, but still failed nevertheless:
Do you want to continue (Y/n)? Y
DEBUG: No bucket specified, retrieving default bucket.
DEBUG: Using bucket [gs://staging.finnsalud.appspot.com].
DEBUG: Service [appengineflex.googleapis.com] is already enabled for project [finnsalud]
Beginning deployment of service [finnsalud-staging]...
DEBUG: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 982, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
resources = command_instance.Run(args)
File "/google-cloud-sdk/lib/surface/app/deploy.py", line 115, in Run
return deploy_util.RunDeploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 669, in RunDeploy
deployer.Deploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 428, in Deploy
source_files = source_files_util.GetSourceFiles(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/source_files_util.py", line 184, in GetSourceFiles
return list(it)
File "/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/util.py", line 165, in FileIterator
entries = set(os.listdir(os.path.join(base, current_dir)))
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/posixpath.py", line 76, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: gcloud crashed (TypeError): expected str, bytes or os.PathLike object, not NoneType
Thinking maybe this was an error relating to the version of my CLI I have also run the following commands to try and update:
gcloud app update
gcloud components update
Unfortunately, this had no change on the output.
I have noticed that when I run this command with the app.yaml env value set to flexible, there are no updates to the logging section on google cloud and no changes to the files uploaded to the project's storage bucket. To me, this indicates that the crash is occurring in the CLI before any communication to the google cloud services is made. If this is correct, then it seems unlikely that the cause of the error would be related to a bad configuration on google cloud and must be related to something (software or configuration) on my local machine.
I have also tried using the 'Hello World' app.yaml configuration on the flexible environments 'Getting Started' page to rule out a configuration error my own application's app.yaml but this also had no change on the output.
Finally, if at any point I change env: flex back to env: standard then the issue does disappear. Unfortunately, as stated above, this won't work for deploying my web sockets feature.
This has gotten me thinking that possibly the error is due to a bug with the gcloud cli application. However, if this were the case, I would have expected to see many more bug reports for this issue by others whom are also using the GAE's flexible environment.
Regardless, given this stack trace points to code within the gcloud cli, I have opened a bug ticket with google which can be found here: https://issuetracker.google.com/issues/176839574
I have also seen this similar SO post, but it is not the exact error I am experiencing and remains unresolved: gcloud app deploy fails with flexible environment
If anyone has any ideas on other steps to try or methods to overcome this issue, I would be immensely grateful if you drop a note on this post. Thanks!
I deployed a nodejs application using the Quickstart for Node.js in the standard environment
Then I changed the app.yaml file from :
runtime: nodejs10
to
runtime: nodejs
env: flex
Everything worked as expected.
It might be related to your specific use case.
Surprisingly, this issue does seem to be related to a bug in the gcloud cli. However, there does seem to be a workaround.
When a --appyaml flag is specified for a deployment to the flex environment, then the CLI crashes with the messages outlines in my question above. However, if you copy your .yaml file renaming to app.yaml (the default) and delete this --appyaml flag when deploying then the build will proceed without errors.
If you have also experienced this error, please follow the google issue as I am working with the google engineers to be sure they reproduce and eventually fix this bug.
Broken app.yaml
runtime:nodejs14
Fixed app.yaml
runtime: nodejs14
I am dead serious. And :
glcoud info --run-diagnostics
was ZERO HELP.
Once I did this the "ERROR: gcloud crashed (TypeError): expected string or bytes-like object" went away.
I guess "colon + space" is part of the spec:
Why does the YAML spec mandate a space after the colon?
Related
Error
My Python app is running fine locally. I've created a Dockerfile and built an image. Inside the app I'm using the Python Google Cloud Logging library. When I try running the image I can see the following error from the Docker container logs:
File "/home/apprunner/app/./app/main.py", line 12, in <module>
2022-12-20 15:14:37 client = google.cloud.logging.Client()
2022-12-20 15:14:37 File "/home/apprunner/app/.venv/lib/python3.9/site-packages/google/cloud/logging_v2/client.py", line 122, in __init__
2022-12-20 15:14:37 super(Client, self).__init__(
2022-12-20 15:14:37 File "/home/apprunner/app/.venv/lib/python3.9/site-packages/google/cloud/client/__init__.py", line 320, in __init__
2022-12-20 15:14:37 _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
2022-12-20 15:14:37 File "/home/apprunner/app/.venv/lib/python3.9/site-packages/google/cloud/client/__init__.py", line 271, in __init__
2022-12-20 15:14:37 raise EnvironmentError(
2022-12-20 15:14:37 OSError: Project was not passed and could not be determined from the environment.
Running the Docker container
I'm running the Docker container using the following commands where I pass in Application Default Credentials:
# Set shell variable
ADC=~/.config/gcloud/application_default_credentials.json \
docker run \
-d \
-v ${ADC}:/tmp/keys/application_default_credentials.json:ro \
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/application_default_credentials.json \
IMAGE_NAME
I'm following the official guide on Docker with Google Cloud Access but using Application Default Credentials instead of a service account.
I've checked that my application_default_credentials.json is present in that location and I've checked that ${ADC} has the correct value:
$ echo $ADC
/Users/ian/.config/gcloud/application_default_credentials.json
Debugging
I see the stack trace points to the line in my code that calls the Logging library:
client = google.cloud.logging.Client()
And below it seems to suggest that it is expecting a project as well as the credentials:
_ClientProjectMixin.__init__(self, project=project, credentials=credentials)
Is this a problem with how I'm passing in my Application Default Credentials or should I be passing in some other project information?
Update
If I explicitly pass in the project argument in my code I can get the Docker container to run successfully:
client = google.cloud.logging.Client(project='my-project')
However I don't want to make code changes for local development and this shouldn't be required. I don't understand why this isn't be pulled out of my ADC(?)
I've been able to get it to run but only by explicitly passing in the project ID.
Solution
The GOOGLE_CLOUD_PROJECT variable is an explicit requirement alongside GOOGLE_APPLICATION_CREDENTIALS. The cleanest way is to pass both in as environment variables when running the container. This is the first place that is searched.
Set shell vars:
ADC=~/.config/gcloud/application_default_credentials.json \
PROJECT=my-project
Docker run:
$ docker run \
-v ${ADC}:/tmp/keys/application_default_credentials.json:ro \
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/application_default_credentials.json \
-e GOOGLE_CLOUD_PROJECT=${PROJECT} \
IMAGE_NAME
Explanation
The docs mention that the project should be inferred from the environment if not explicitly provided:
# if project not given, it will be inferred from the environment
client = google.cloud.logging.Client(project="my-project")
To be inferred from the environment you need to have:
Installed the Google Cloud SDK
Created Application Default Credentials (ADC).
gcloud auth application-default login
Set an active project.
gcloud config set project PROJECT_ID
Passing ADC in to the locally running container through environment variables works for authentication but it doesn't pass in the active project as this is set in your local configuration (3)(~/.config/gcloud/configurations on Mac/Linux). So no project can be inferred from the environment inside the container as it is not set and not passed. So it searches through the list of locations in order and doesn't find anything.
Best Practice
It's good practice to pass both authentication credentials and project identifier from the same place:
Credentials and project information must come from the
same place (principle of least surprise).
Be explicit in setting them rather than relying on Application Default Credentials:
we really encourage people to explicitly pass credentials and
project to the client constructor and not depend on the often surprising behavior of application default credentials.
Make the easy to find by setting them in the first place that will be searched:
GOOGLE_CLOUD_PROJECT environment variable
GOOGLE_APPLICATION_CREDENTIALS JSON file
With these in mind, passing them both in as environment variables ticks all the boxes.
Note: This looks to be the same throughout the Google Cloud Python Client Library and not just in Logging.
I'm having some problems with retrieving job output from an AWS glacier vault.
I initiated a job (aws glacier initiate-job), the job is indicated as complete via aws glacier, and then I tried to retrieve the job output
aws glacier get-job-output --account-id - --vault-name <myvaultname> --job-id <jobid> output.json
However, I receive an error: [Errno 2] No such file or directory: 'output.json'
Thinking that perhaps the file needed be created first, and if i did create the file first, (which really doesn't make sense), one would receive the [Errno 9] Bad file descriptor error.
I'm currently using the following version of the AWS CLI:
aws-cli/2.4.10 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
I tried using the aws CLI from both an Administrative and non-Administrative command prompt with the same result. Any ideas on making this work?
From a related reported issue you can try run this command in a DOS window::
copy "c:\Program Files\Amazon\AWSCLI\botocore\vendored\requests\cacert.pem" "c:\Program Files\Amazon\AWSCLI\certifi"
It seems to be an certificate error
sorry if my question is too basic, but cannot solve it.
I am experimenting with mlflow currently and facing the following issue:
Even if I have set the tracking_uri, the mlflow artifacts are saved to the ./mlruns/... folder relative to the path from where I run mlfow run path/to/train.py (in command line). The mlflow server searches for the artifacts following the tracking_uri (mlflow server --default-artifact-root here/comes/the/same/tracking_uri).
Through the following example it will be clear what I mean:
I set the following in the training script before the with mlflow.start_run() as run:
mlflow.set_tracking_uri("file:///home/#myUser/#SomeFolders/mlflow_artifact_store/mlruns/")
My expectation would be that mlflow saves all the artifacts to the place I gave in the registry uri. Instead, it saves the artifacts relative to place from where I run mlflow run path/to/train.py, i.e. running the following
/home/#myUser/ mlflow run path/to/train.py
creates the structure:
/home/#myUser/mlruns/#experimentID/#runID/artifacts
/home/#myUser/mlruns/#experimentID/#runID/metrics
/home/#myUser/mlruns/#experimentID/#runID/params
/home/#myUser/mlruns/#experimentID/#runID/tags
and therefore it doesn't find the run artifacts in the tracking_uri, giving the error message:
Traceback (most recent call last):
File "train.py", line 59, in <module>
with mlflow.start_run() as run:
File "/home/#myUser/miniconda3/envs/mlflow-ff56d6062d031d43990effc19450800e72b9830b/lib/python3.6/site-packages/mlflow/tracking/fluent.py", line 204, in start_run
active_run_obj = client.get_run(existing_run_id)
File "/home/#myUser/miniconda3/envs/mlflow-ff56d6062d031d43990effc19450800e72b9830b/lib/python3.6/site-packages/mlflow/tracking/client.py", line 151, in get_run
return self._tracking_client.get_run(run_id)
File "/home/#myUser/miniconda3/envs/mlflow-ff56d6062d031d43990effc19450800e72b9830b/lib/python3.6/site-packages/mlflow/tracking/_tracking_service/client.py", line 57, in get_run
return self.store.get_run(run_id)
File "/home/#myUser/miniconda3/envs/mlflow-ff56d6062d031d43990effc19450800e72b9830b/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py", line 524, in get_run
run_info = self._get_run_info(run_id)
File "/home/#myUser/miniconda3/envs/mlflow-ff56d6062d031d43990effc19450800e72b9830b/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py", line 544, in _get_run_info
"Run '%s' not found" % run_uuid, databricks_pb2.RESOURCE_DOES_NOT_EXIST
mlflow.exceptions.MlflowException: Run '788563758ece40f283bfbf8ba80ceca8' not found
2021/07/23 16:54:16 ERROR mlflow.cli: === Run (ID '788563758ece40f283bfbf8ba80ceca8') failed ===
Why is that so? How can I change the place where the artifacts are stored, this directory structure is created? I have tried mlflow run --storage-dir here/comes/the/path, setting the tracking_uri, registry_uri. If I run the /home/path/to/tracking/uri mlflow run path/to/train.py it works, but I need to run the scripts remotely.
My endgoal would be to change the artifact uri to an NFS drive, but even in my local computer I cannot do the trick.
Thanks for reading it, even more thanks if you suggest a solution! :)
Have a great day!
This issue was solved by the following:
I have mixed the tracking_uri with the backend_store_uri.
The tracking_uri is where the MLflow related data (e.g. tags, parameters, metrics, etc.) are saved, which can be a database. On the other hand, the artifact_location is where the artifacts (other, not MLflow related data belonging to the preprocessing/training/evaluation/etc. scripts).
What led me to mistakes is that by running mlflow server from command line one should set up for the --backend-store-uri the tracking_uri (also in the script by setting the mlflow.set_tracking_uri()) and for --default-artifact-location the location of the artifacts. Somehow I didn't get that the tracking_uri = backend_store_uri.
Here's my solution
Launch the server
mlflow server -h 0.0.0.0 -p 5000 --backend-store-uri postgresql://DB_USER:DB_PASSWORD#DB_ENDPOINT:5432/DB_NAME --default-artifact-root s3://S3_BUCKET_NAME
Set the the tracking uri to an HTTP URI like
mlflow.set_tracking_uri("http://my-tracking-server:5000/")
I am trying to deploy a sample python app which I got from another tutorial. However, the deployment fails as below:
gcloud app deploy Beginning deployment of service [default]... ERROR:
gcloud crashed (FileNotFoundError): [Errno 2] No such file or
directory:
'/Users/nileshdeshmukh/Desktop/Training/Python/FlaskIntroduction-master/env/.Python'
My app.yaml file is as below:
runtime: python3
env: standard
runtime_config:
python_version: 3
I have all dependencies copied in env/bin but the build process is looking for env only..
I think the problem would be solved if the deployment process looks at env/bin, but don't know how to force it to look at given path
The runtime_config setting is for App Engine flex only and isn't needed for App Engine Standard. You can safely remove it.
As per the error, you should ensure that all your dependencies are self-contained and shipped with your app or listed in your requirements.txt file.
Be careful, some gcloud commands use .gitignore file to prevent sending useless file to Cloud for building your app.
You can override this behavior by creating a .gcloudignore file. Same syntax as git ignore but take into account only by gcloud commands and not by git. By the way you can differentiate the file to send to the cloud and file to send to git
I am trying to use gcloud in order to deploy a nodejs app. I have followed this tutorial.
When I run:
gcloud preview app deploy
I have 2 possible issue:
First :
Beginning deployment... If this is your first deployment, this may
take a while...failed. ERROR: gcloud crashed (ResponseNotReady)
Or sometimes the process go further :
Verifying that Managed VMs are enabled and ready. ERROR: gcloud
crashed (ResponseNotReady)
The log look like that :
File "C:\Program Files (x86)\Google\Cloud
SDK\google-cloud-sdk\bin..\lib\third_party\httplib2__init__.py",
line 1308, in _conn_request
response = conn.getresponse()
File "C:\python27_x64\lib\httplib.py", line 1119, in getresponse
raise ResponseNotReady() ResponseNotReady 2016-02-22 15:13:00,937 ERROR root gcloud crashed (ResponseNotReady):
Thank you !
Seems to be some connection instability... I made it without any change but retrying like an hundred times.