Is there a Python3 library which will determine if we are running on GCP or another cloud architecture as opposed to a native architecture?
Something like platform.platform() or jaraco.docker.is_docker(), but for the cloud?
Yes, you can. Each cloud vendor provides unique interfaces and usually environment variables, system services, etc that can be used to detect the cloud vendor. For Google, I use a simple method of connecting to the Metadata server for compute services.
For Python look at libraries such as cloud-detect. This is not a recommendation, just a link.
cloud-detect
No, simply because the Python codes actually running in a virtual machine in the cloud.
Related
I want my C++ application to launch an arbitrary app (let's say a python script through a python interpreter) inside a secure enclave (Intel SGX). Is that even possible?
The steps are the following.
My app initializes an enclave and performs its attestation.
Next, it somehow uploads a python interpreter and a python script to the enclave.
It also uplaods to the enclave some piece of data to be processed by the script.
Then, the script is being launched inside the enclave and the data is being processed.
Finally, the processing result is uploaded back to the host.
Is this scenario possible? If yes, are there any examples on how to do so?
Microsoft OpenEnclave is also a choice.
There are many examples of adding unmodified libraries to SGX and then run the toy applications: oeapkman, Package Manager and Toolbox for enclave development: apkman.
If AWS Nitro Enclaves would satisfy you then the Oblivious framework let’s you do what you are describing.
There is a full tutorial and YouTube walkthrough of deploying FastAPI servers, as an example, here.
Disclosure: I work with Oblivious but this post is in no way an ad or plug, I think it just does what #pgr is asking for.
I am trying to run a python3 program continuously on GCP. What is the best way to do this?
So far I have tried using a google compute engine virtual machine running Debian linux. I used nohup but it still hangs up when the ssh connection is broken.
What other ways could I try to run the program through the vm? Are there better alternatives using GCP to run the program continuously?
Python's installation depend on different operating systems, documentation [1] could help you to run python program in Linux or in Windows without any trouble.
On the other hand, Google App Engine applications [2] are easy to create, easy to maintain, and easy to scale as your traffic and data storage needs change. With App Engine, there are no servers to maintain. You simply upload your application and very easy to operate. The documentation [3] also could be very helpful for you.
To know more about python on Google Cloud Platform, please have the documentation [4].
[1] https://cloud.google.com/python/setup#installing_python
[2] https://codelabs.developers.google.com/codelabs/cloud-app-engine-python3/#0
[3] https://cloud.google.com/python/getting-started
[4] https://cloud.google.com/python
We created and tested several Azure Function Apps hosted at Windows. While creating new Azure Function App in what scenario do I select OS other than Windows? Meaning Linux or Docker.
I created test instances for all three OS selection options and basic settings of each of them appear to be very close.
Linux or Docker is useful if your functions have dependencies that only work on Linux/Docker. For example, some node.js native libraries only work on Linux, and will never work on Windows.
If you don't need Linux for anything specific, then I suggest sticking to Windows since that is currently (at the time of writing) the best and most supported environment for running Azure Functions.
Azure Functions 2.0 runtime is based on .NET Core, so it is cross-platform. If you choose Linux/Docker, Functions runtime will be deployed on Linux.
2.0 is still in preview, so Linux/Docker are not supported in production yet. For now, Consumption Plan (pay per call) is not supported.
See The Azure Functions on Linux Preview. Quote:
Functions on Linux can be hosted in a dedicated App Service tier in 2 different modes:
You bring the Function App code and we provide and manage the container, no specific Docker related knowledge required.
You bring your own Docker container including the Azure Functions runtime 2.0, specific dependencies, and Function App code.
For consumptions mode, the cold start varies a little bit among the OS.
It looks like, although the average time is very close between Windows and Linux, the best and worst cases are much better for Linux... which kind of makes sense.
Check this as a good reference: https://mikhail.io/serverless/coldstarts/azure/
Now, if you are deploying to a dedicated Apps Service Plan, it plays a bigger role. Linux Plans are cheaper than Windows Plans due to the OS licensing cost.
My lab just got a sponsorship from Microsoft Azure and I'm exploring how to utilize it. I'm new to industrial level cloud service and pretty confused about tons of terminologies and concepts. In short, here is my scenario:
I want to experiment the same algorithm with multiple datasets, aka data parallelism.
The algorithm is implemented with C++ on Linux (ubuntu 16.04). I made my best to use static linking, but still depends on some dynamic libraries. However these dynamic libraries can be easily installed by apt.
Each dataset is structured, means data (images, other files...) are organized with folders.
The idea system configuration would be a bunch of identical VMs and a shared file system. Then I can submit my job with 'qsub' from a script or something. Is there a way to do this on Azure?
I investigated the Batch Service, but having trouble installing dependencies after creating compute node. I also had trouble with storage. So far I only saw examples of using Batch with Blob storage, with is unstructured.
So are there any other services in Azure can meet my requirement?
I somehow figured it out my self based on the article: https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-classic-hpcpack-cluster/. Here is my solution:
Create a HPC Pack with a Windows head node and a set of Linux compute node. Here are several useful template in Marketplace.
From Head node, we can execute command inside Linux compute node, either inside HPC Cluster Manager, or using "clusrun" inside PowerShell. We can easily install dependencies via apt-get for computing node.
Create a File share inside one of the storage account. This can be mounted by all machines inside the cluster.
One glitch here is that for some encryption reason, you can not mount the File share on Linux machines outside the Azure. There are two solutions in my head: (1) mount the file share to Windows head node, and create file sharing from there, either by FTP or SSH. (2) create another Linux VM (as a bridge), mount the File share on that VM and use "scp" to communicate with it from outside. Since I'm not familiar with Windows, I adopted the later solution.
For executable, I simply uploaded the binary executable compiled on my local machine. Most dependencies are statically linked. There are still a few dynamic objects, though. I upload these dynamic object to the Azure and set LD_LIBRARY_PATH when execute programs on computing node.
Job submission is done in Windows head node. To make it more flexible, I wrote a python script, which writes XML files. The Job Manager can load these XML files to create a job. Here are some instructions: https://msdn.microsoft.com/en-us/library/hh560266(v=vs.85).aspx
I believe there should be more a elegant solution with Azure Batch Service, but so far my small cluster runs pretty well with HPC Pack. Hope this post can help somebody.
Azure files could provide you a shared file solution for your Ubuntu boxes - details are here:
https://azure.microsoft.com/en-us/documentation/articles/storage-how-to-use-files-linux/
Again depending on your requirement you can create a pseudo structure via Blob storage via the use of containers and then the "/" in the naming strategy of your blobs.
To David's point, whilst Batch is generally looked at for these kind of workloads it may not fit your solution. VM Scale Sets(https://azure.microsoft.com/en-us/documentation/articles/virtual-machine-scale-sets-overview/) would allow you to scale up your compute capacity either via load or schedule depending on your workload behavior.
My developer has written a web scraping app on Linux on his private machine, and asked me to provide him with a Linux server. I setup an account on Google Compute Engine, created a Linux image with enough resources and a sufficiently large SSD drive. Three weeks later he is claiming that working on Google is too complex quote - "google is complex because their deployment process is separate for all modules. especially i will have to learn about how to set a scheduler and call remote scripts (it looks they handle these their own way)."
He suggests I create an account on Hostgator.com.
I appreciate that I am non-technical, but I cannot be that difficult to use Linux on Google?! Am I missing something? Is there any advice you could give me?
Regarding the suggestion to create an account on Hostgator to utilize what I presume would be a VPS in lieu of a Virtual Machine on GCE , I would suggest seeking a more concrete example from the developer.
For instance, the comment about the "scheduler", let's refer to it as some process that needs to execute on a regular basis:
How is this 'process' currently accomplished on the private machine ?
How would it be done on the VPS ?
What is preventing this 'process' from being done on the GCE VM ?