My app needs cmake, libx11-dev and libpng-dev to build. I came across this documentation, which leads me to believe that I can list these as dependencies for my app to run on the Google App Engine platform, although I cannot figure out how. I was successfully able to run my app in a Compute Engine instance, although this is costly and, if I'm not mistaken, unnecessary. How do I get the packages listed at the beginning of the question installed beyond session end?
You can only list Node.js dependencies that way. From Declaring and managing dependencies (emphasis mine):
You can use any Linux-compatible Node.js package with App Engine
flexible environment, including packages that require native (C)
extensions.
You can use dependencies other than Node.js (at least cmake in your list) but only in the flexible environment, via a custom runtime. From About Custom Runtimes:
Custom runtimes allow you to define new runtime environments, which
might include additional components like language interpreters or
application servers.
See also Building Custom Runtimes.
You need to keep in mind that the App Engine Flexible Environment still uses Compute Engine instances so may not get an additional benefit from moving across to this
Based on Google Compute Engine, the App Engine flexible environment
automatically scales your app up and down while balancing the load.
The issue that you have is that if you require cmake, libx11-dev and libpng-dev to build your application you'll still need to use an underlying Compute Engine VM in order to run the application. This will be the case even if you consider moving across to Kubernetes Engine as well.
If you're looking to manage costs for your application, perhaps consider downsizing the VM to a smaller instance or look into modifying your application to suit the App Engine Standard Environment or use Cloud Functions
Related
Why do I need to bundle runtime environment with every Electron app and punish the ones trying to download the app? Can't it create a ClickOnce app like thing where the runtime environment is downloaded if it is not available already or download additional dependencies as necessary and link dynamically? It will save up on a lot of storage space worldwide I am sure.
My Node application needs to be deployed on Windows and Linux. The main deployment package is built on a Linux CI server.
When this package is deployed to Windows, it crashes immediately due to missing native bindings, such as those for sqlite. Only the bindings for the build platform (Linux) are restored.
With a deadline approaching, we just set up a Windows build configuration which outputs a Windows specific package that contains the appropriate bindings, and we choose the appropriate artifact to bundle in the installer.
This works but feels fragile, as we would need to keep the Node versions in sync between the two otherwise unrelated environments. I would like to be able to do this with a single build configuration.
I couldn't find any guidance on how this is done. I'm imagining a command-line option like --platform=windows to npm ci, or a modification to package.json but I couldn't find any information about this. Presumably this is a reasonably rare requirement, and perhaps there is no tooling around this, which would be a shame.
Another requirement is that the application must be installed without an internet connection. We cannot run npm ci or npm install when we install it as some of our clients do not permit their servers to access the public internet.
Based on your requirements it sounds like building a package on each required platform would be the safest bet, with the least number moving parts to go wrong.
As the comments have suggested most projects rely on an npm install on the required platform so you are stepping into not that common territory.
This works but feels fragile, as we would need to keep the Node versions in sync between the two otherwise unrelated environments. I would like to be able to do this with a single build configuration.
Node uses NODE_MODULE_VERSION (displayed on the releases page) to track ABI compatibility for native modules. This only changes with a new major Node release number.
The CI builds would need to create app packages for each major version of Node you run on each platform. Keeping the Node.js major versions in sync for the application a good thing in any case. Running Node N and N-1 builds until that can be achieved is good cover and probably the best option with the air gap requirements.
NPM Cache
If the air gapped clients are largely on common networks, an NPM cache/proxy (nexus/verdaccio) may be of use. The NPM cache will need a process to snapshot the repo after a production npm install on all required platforms, to be pushed out to your endpoints. Unfortunately binary modules are often distributed out of band from NPM so won't be stored in regular NPM caches. Each client instance will need a complete build environment to build any native modules from source which can sometime present it's own difficulties on Windows platforms.
Alternatives
Node.js is not a great platform for distributing packaged applications to many diverse clients, especially if you need to distribute Node itself. Any language with an external VM requirement presents difficulties. Nodes package management choices and reliance on native modules exacerbate this.
I've given up in the past and converted clients (albeit thin) to Go, as it lends itself to cross platform distribution a lot better by removing the external runtime requirement and having less variables.
I want to create a Python script that will parse 40.000 PDF files(text and images). Since I saw that there is no easy method to check if a page contains images I think I should use textract module.
Ideally I would deploy to Google App Engine.
My question is, for textract I've also installed other packages beside Python to my system. Can I deploy the script(with proper requirements.txt file) on Google Cloud App Engine without problem? or I will to use something else?
It is possible to use App Engine, but only with the Flexible environment and using a custom runtime, which allows you to add non-python dependencies (and also python dependencies not installable via pip):
Custom runtimes allow you to define new runtime environments, which
might include additional components like language interpreters or
application servers.
See also Building Custom Runtimes.
I am using Google App Engine to run my NodeJS app on flexible env, now i wanted to generate FlameGraph but the thing is as App Engine itself handle scaling and deploying of instances now can anyone please tell me how can i generate Flamegraph(NodeJs Profiling) to trace Requests coming on my NodeJs server.
If anyone of you has worked on Google App Engine on any Framework(NodeJs or any other), Can you all please tell me how did you solve this kind of problem on App Engine.
Update -
Why We need to delete the instance after debugging it.
Flame graphs are a visualization of profiled software, allowing the
most frequent code-paths to be identified quickly and accurately.
So FlameGraphs have nothing to do with networking, scaling or deploying to GCP.
Anyhow, FlameGraph is a just a 3rd party tool you can install and run. So the answer is you can make it work same way you would install and run on your local computer.
If you don't know how to use FlameGraph to profile NodeJS, then you should start reading some tutorials, as this site is not for that type of questions. A good one is here: https://nodejs.org/en/blog/uncategorized/profiling-node-js/
UPDATE: How to ssh into app engine flex instance
In google cloud console go to App Engine Flex -> Instances
After installing Google Plugin for Eclipse, there are options to create three different types of projects
Google App Engine Flexible Java Project
Google App Engine Standard Java Project
Google Dataflow Java Project
What is the difference between these three?
App Engine allows you to deploy application with minimum infrastructure related configuration. Standard is quicker in both deploying and scaling your application, but has a limited choice of languages and libraries to use. Flexible manages docker containers so it can run almost anything and has more powerful machines to chose from, same that Compute Engine has, but it takes longer to deploy and is in general more expensive.
Lightweight generic code that should be always available should run on Standard. A batch job that is processed once every day and requires an expensive setup and third party libraries would better run on Flex.
More on that here
Google App Engine Dataflow Java Project
It's not an App Engine, it says:
Google App Engine Dataflow Java Project
It's a service directed specifically at batch jobs and streaming. More on it here.