If I have 5 bots for trading and a along with this a script that does some updating on prices using scraping. All these files uses Node js. Now, I was able to deploy all the 6 scripts on digital ocean, but due to 6 scripts running together as 6 different processes the CPU usage in even their most expensive plan became 100%. Then I decided to shift to google cloud. But it turns out with GPU it is hell expensive.
Essentially what I want to do is that run the 6 scripts at 3 distinct times in a day for 10 mins. Other than those particular times the 6 scripts do nothing.
I have set a file named concurrently.js that runs all these scripts using the command concurrently.
Is it possible to run concurrently.js at 3 particular times of the day and then after 10 mins when the job is done, shut down the virtual machine?
Say machine turns on at 12.00pm then the 6 files work for 10 mins and then the machine shuts off at 12.10 pm. And then turns on at say 3.05 pm and so on.
If I can schedule on and off of the VM I can afford google cloud.
I got to know about cron and google cloud scheduler, but they need an App url to schedule tasks. But I don't have an app url because I don't have app only, I just want to run the concurrently.js file present in the virtual machine along with other files, can I do the scheduling?
Any help is highly appreciated!!!
You can do this with Google Cloud. Here the process
Cloud Scheduler start your Compute Engine VM
At startup, the Compute Engine VM runs a startup script that run your process
At the end of the process the VM auto shutdown
So for that you need to
Call the Compute Engine start API
Set a startup script on your VM
Shutdown the VM automatically at the end of the processing
If you are stuck in one step, let me know, I could narrow my help.
Related
I'm currently running 2 scripts on a weekly schedule on a raspberry pi with the following configuration:
Cron executes a python script at a fixed time weekly. This python script waits between 0 and 50 hours then runs python script A. It waits about 16 hours and runs script A again 3 more times every 8 hours (The script takes about 4x longer to run the first time). 8 hours after the 4th run it runs script B.
I would like to move my scripts to Google Cloud VM for improved reliability but running the VM 24/7 just to run 30 hours worth of computations over a 100 hour period is inefficient and expensive.
I know I can use Google Scheduler as my cron to initiate the VM weekly but I still risk letting it run up to 50 hours waiting for script A to run. I understand cron supports adding a random sleep interval as listed in the example here:
30 8-21/* * * * sleep ${RANDOM:0:2}m ; /path/to/script.php
However, from what I've discovered, Google Cloud Scheduler is limited to 60 minutes and rightfully so. In this case what are my options? Does Google Cloud Task support delayed triggering of VM (up to 50 hours)? Is this something Pub Sub would support instead?
My scripts use a python library that I don't think is compatible with Google App Engine so I would further need to figure how to trigger a specific script in the VM on trigger.
You can use Cloud Scheduler and Pub/Sub to trigger a Cloud Function that will start your VM and execute your script. If you do not want your Compute Engine instances to be running 24/7, at the end of your script you can have your Cloud Function stop your VM.
You can find how to schedule compute instances with Cloud Scheduler here and how to use HTTP functions in Cloud Functions to start and stop your Compute Engine instance [1].
Most importantly, here is the documentation on how to use Cloud Scheduler and Pub/Sub to trigger a Cloud Function [2].
[1] https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule
[2] https://cloud.google.com/scheduler/docs/tut-pub-sub
[3] Cloud Functions: https://cloud.google.com/functions/docs/concepts/overview
I need an Azure VM (Ubuntu) to do some task (java application) every 10 minutes. Because the task lasts usually less than a minute I would save money if could start the machine every 10 minutes and stop it when the task accomplishes. I learned that I can schedule start and stop times in automation account, but more optimal would be to stop the VM in the very moment that task is completed. Is there a simple way to do that?
This really sounds like a job for Azure Batch. If you are looking for an IaaS solution, Azure Batch will do the job for you. Have a look at it: https://azure.microsoft.com/en-gb/services/batch/#overview.
It allows you to use VM's with your preferred OS (in Azure Batch it is called a node), and run a set of tasks. Once finished, the VM will be de-allocated.
So each node runs a set of pools, in each pool you have a job, and in each job you can have tasks. A task can be for example a cmd line that runs a specific app. So for instance you could just run example.exe 1 2 on a windows OS or the equivalent command line for an Ubuntu OS.
The power here is that it will allocate the tasks to run on the VM when you add them to the job, and then the VM will be disposed off once finished, and you would only pay for the compute time.
The disadvantages of this is that it is a stateless VM, therefore anything that you need installing or storing you would have to use alternative methods. Azure Batch allows you to pre-install a program (for example your Java application) each time it initiates. Also if you are using files and/or expecting files to be created, you would need a blob storage to support this. So if you are expecting it to use a certain amount of files, store them on blob storage and then write back to the blob storage if your program is doing this.
Finally your scheduler, this really depends on how you want to deal with it, if you have a local server or a server on Azure that is already running 24/7 you can add a scheduled job to the scheduler and run a program that will add the task to the Azure Batch. Or if you don't mind using Azure Functions, you can just add a timer Azure Function that will add a task to the job. There are multiple ways of dealing with this, you may already have an existing solution.
Hope you find this useful!
I have a python code which reads data from one cloud system via rest api using the requests module and then writes data back to another cloud system via rest api . This code runs anywhere from 1 to 4 hours every week. Is there a place in Google Cloud Platform , I can execute this code on a periodic basis. Sort of like a scheduled batch job . Is there a serverless option to do this in App Engine . I know about the App engine cron service but seems like it is only for calling a URL regularly . Any thoughts ? Appreciate your help.
Google Cloud Scheduler could be the tool you are looking for. As it is mentioned in its documentation:
Cloud Scheduler is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, cloud infrastructure operations, and more. You can automate everything, including retries in case of failure to reduce manual toil and intervention.
Here you have the quickstart for Cloud Scheduler, and also another tutorial for Cron jobs.
You can use the Google Genomics API pipelines.run endpoint to run a long-running job on a Google Compute Engine virtual machine and then it will destroy the machine when it's done. If your job will run for less than 24 hours and it can handle a failure, then you can use a Preemptible VM to save cost.
Pipelines: Run
https://cloud.google.com/genomics/reference/rest/v2alpha1/pipelines/run
Preemptible Virtual Machines
https://cloud.google.com/preemptible-vms/
You could use Cloud Scheduler to kick off the job
Pipelines may be preferred to trying to use one of the serverless technologies because they don't tend to handle the long running jobs as well.
You can use AI Platform Training to run any arbitrary Python package — it doesn’t have to be a machine learning job.
I have 2 azure vm's created, 1 windows server 2012 r2, and 1 Ubuntu 14.
It takes both those vm's approximately 5 minutes to startup.
Is there a way I can speed up the process?
I don't need the vm's running continuously, I prefer to start/stop, as per need.
There are no steps you can take to speed up the VM start. Resources must be allocated and the VM provisioned.
What I can recommend is setting up a script to auto start / stop your VM's based on a schedule. For example, if you are using them in a class room environment you can set it up so that they start early (6am?) and shut down each day by 5pm.
You can find some more information about this here.
I'm running a Node.js app in an Azure website and I'm trying to get a timer to go off every 30 seconds indefinitely. I've tried using both setInterval and node-cron but both of these stop working after about half an hour. There are no errors in the logs, and the site stays up, but the timer stops firing. I can start or recreate the timer again but it just stops working after another half an hour.
I am on the free trial period, am I running into some sort of account restriction here? Or is there some other way I should be doing this or some way to work out what is making the timer stop?
When hosting Node.js on a website (or in fact using the node.js powershell support for Cloud Services) your node process is hosted by IISNode.
IISNode will manage the lifecycle of the node process. I'm guessing it's intentionally shutting you down after 30 minutes.
You need to run node yourself in Azure if you want a process that's going to stay up. There are two options:
Use the Virtual Machines (currently in preview), and manually copy node.exe, and your javascript onto a virtual machine, and start your app manually (or as a scheduled task or something). You can use either Windows or Linux.
Use Cloud Services, and create a Worker Role which starts node as a startup task. This gives you the ability to scale as well.